With the recent rise in opioid overdose and death1<br/><br/>, chronic opioid therapy (COT) programs using<br/>Center of Disease Control (CDC) guidelines have been implemented across the United States8<br/>.<br/>Primary care clinicians at Mayo Clinic initiated a COT program in September of 2017, during the<br/>use of Cerner Electronic Health Record (EHR) system. Study metrics included provider<br/>satisfaction and perceptions regarding opioid prescription. Mayo Clinic transitioned its EHR<br/>system from Cerner to Epic in October 2018. This study aims to understand if provider perceptions<br/>about COT changed after the EHR transition and the reasons underlying those perceptions.
Methods: The standard NLP process was used for this study in which a gold standard was reached through matched paired annotations of the forum text in brat and a neural network was trained on the content. Following the annotation process, adjudication occurred to increase the inter-annotator agreement. Categories were developed by local physicians to describe the questions and three pilots were run to test the best way to categorize the questions.
Results: The inter-annotator agreement, calculated via F-score, before adjudication for a 0.7 threshold was 0.378 for the annotation activity. After adjudication at a threshold of 0.7, the inter-annotator agreement increased to 0.560. Pilots 1, 2, and 3 of the categorization activity had an inter-annotator agreement of 0.375, 0.5, and 0.966 respectively.
Discussion: The inter-annotator agreement of the annotation activity may have been low initially since the annotators were students who may have not been as invested in the project as necessary to accurately annotate the text. Also, as everyone interprets the text slightly differently, it is possible that that contributed to the differences in the matched pairs’ annotations. The F-score variation for the categorization activity partially had to do with different delivery systems of the instructions and partially with the area of study of the participants. The first pilot did not mandate the use of the original context located in brat and the instructions were provided in the form of a downloadable document. The participants were computer science graduate students. The second pilot also had the instructions delivered via a document, but it was strongly suggested that the context be used to gain an understanding of the questions’ meanings. The participants were also computer science graduate students who upon a discussion of their results after the pilot expressed that they did not have a good understanding of the medical jargon in the posts. The final pilot used a combination of students with and without medical background, required to use the context, and included verbal instructions in combination with the written ones. The combination of these factors increased the F-score significantly. For a full-scale experiment, students with a medical background should be used to categorize the questions.
further process output from a machine learning based named entity recognition (NER) tool for the purposes of (1) linking references to radiological images with the corresponding clinical findings and (2) extracting primary and incidental findings.
Methods: The project’s system utilized a regular expression to extract image references. All CTPA reports were first processed with NER software to obtain the text and spans of clinical findings. A heuristic was used to determine the appropriate clinical finding that should be linked with a particular image reference. Another regular expression was used to extract primary findings from NER output; the remaining findings were considered incidental. Performance was
assessed against a gold standard, which was based upon a manually annotated version of the CTPA reports used in this project.
Results: Extraction of image references achieved a 100% accuracy. Linkages between these references and exact gold standard spans of the clinical findings achieved a precision of 0.24, a recall of 0.22, and an F1 score of 0.23. Linkages with partial spans of clinical findings as determined by the gold standard achieved a precision of 0.71, a recall of 0.67, and an F1 score of 0.69. Primary and incidental finding extraction achieved a precision of 0.67, a recall of 0.80, and
an F1 score of 0.73.
Discussion: Various elements reduced system performance such as the difficulty of exactly matching the spans of clinical findings from NER output with those found in the gold standard. The heuristic linking clinical findings and image references was especially sensitive to NER false positives and false negatives due to its assumption that the appropriate clinical finding was that which was immediately prior to the image reference. Although the system did not perform as well as hoped, lessons were learned such as the need for clear research methodology and proper gold standard creation; without a proper gold standard, problem scope and system performance cannot be properly assessed. Improvements to the system include creating a more robust heuristic, sifting NER false positives, and training the NER tool used on a dataset of CTPA reports.
Research Objective Social determinants of health (SDOH) are the conditions in one’s living environment that affect health, functioning, and quality of life. Total joint arthroplasty (TJA) is a surgical procedure to replace a damaged joint with an artificial joint. TJA complications include acute myocardial infarction, pneumonia, sepsis, surgical site bleeding, pulmonary embolism, or periprosthetic joint infection. Previous research demonstrates that Black race, Hispanic ethnicity and poverty were negatively associated with TJA outcomes in veterans. The goal of this mixed methods quality improvement study is to determine if SDOHs affect TJA complications at a health system in the Phoenix metropolitan area. Methodology For this study, records from patients who underwent hip or knee TJAs at any of the four system facilities between 2/2019-2/2020 were included. Demographics and clinical data were extracted from the electronic health record (EHR) via Midas+ Care Management with SDOH variables from case manager notes corresponding to food, utilities, housing and transportation insecurities, and interpersonal safety. Complications were identified using ICD-10 codes. SDOH for individuals with and without complications were compared. A multinomial logistic regression was performed in SPSS to identify significant variables. Semi-structured interviews with case managers (n=2), orthopedic surgeons(n=5), and primary care physicians (n=4) were performed to explore care team interactions with SDOH. Interview notes were coded and analyzed based on response frequency and themes. Results Of 2,520 patients who underwent TJA, 50 (1.98%) experienced a TJA complication. Of those, 38% screened positive for an SDOH. For those without a TJA complication, 27% screened positive for an SDOH (p=0.093). Most interview participants identified a correlation between socioeconomic status and surgical outcomes. They also recognized that language barriers for Spanish-speaking individuals and family involvement post-discharge are significant factors in TJA outcomes. Conclusions This single system mixed methods retrospective quality improvement study demonstrates that patients who screen positive for an SDOH are more likely to experience a TJA complication. We recommend that SDOH assessments be obtained for all patients undergoing TJA, be available to care teams, and be incorporated into care plans to improve outcomes.