The Medical Scribe: Corpus Development and Model Performance Analyses

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 2036–2044 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 2036 The Medical Scribe: Corpus Development and Model Performance Analyses Izhak Shafran, Nan Du, Linh Tran, Amanda Perry, Lauren Keyes, Mark Knichel, Ashley Domin, Lei Huang, Yuhui Chen, Gang Li, Mingqiu Wang, Laurent El Shafey, Hagen Soltau, Justin S. Paul Google Inc. Abstract There is a growing interest in creating tools to assist in clinical note generation using the audio of provider-patient encounters. Motivated by this goal and with the help of providers and medical scribes, we developed an annotation scheme to extract relevant clinical concepts. We used this annotation scheme to label a corpus of about 6k clinical encounters. This was used to train a state-of-the-art tagging model. We report ontologies, labeling results, model performances, and detailed analyses of the results. Our results show that the entities related to medications can be extracted with a relatively high accuracy of 0.90 F-score, followed by symptoms at 0.72 F-score, and conditions at 0.57 F-score. In our task, we not only identify where the symptoms are mentioned but also map them to canonical forms as they appear in the clinical notes. Of the different types of errors, in about 19-38% of the cases, we find that the model output was correct, and about 17-32% of the errors do not impact the clinical note. Taken together, the models developed in this work are more useful than the F-scores reflect, making it a promising approach for practical applications. Keywords: Medical Scribe, Information Extraction, Clinical Encounters, Span-Attribute Tagging (SAT) Model 1. Introduction Medical providers across the United States are required to document clinical visits in the Electronic Health Records. This need for documentation takes up a disproportionate amount of their time and attention, resulting in provider burnout (Wachter and Goldsmith, 2018; Xu, 2018). One study found that full-time primary care physicians spent about 4.5 hours of an 11-hour workday interacting with the clinical documentation systems, yet were still unable to fin- ish their documentation and had to spend an additional 1.4 hours after normal clinical hours (Arndt et al., 2017). Speech and natural language processing are now suffi- ciently mature that there is considerable interest, both in academia and industry, to investigate how these technolo- gies can be exploited to simplify the task of documenta- tion, and to allow providers to dedicate more time to pa- tients. While domain-specific automatic speech recognition (ASR) systems that allow providers to dictate notes have been around for a while, recent work has begun to address the challenges associated with generating clinical notes di- rectly from speech recordings. This includes inducing topic structure from conversation data, extracting relevant infor- mation, and clinical summary generation (Quiroz et al., 2019). In one recent work, authors outlined an end-to-end system; however, the details were scant without empirical evaluations of their building blocks (Finley et al., 2018a). One of the simplistic approaches uses a hand crafted …

Content truncated to 500 words