The USMLE® Step 2 Clinical Skills Patient Note Corpus

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2880 - 2886 July 10-15, 2022 ©2022 Association for Computational Linguistics The USMLE® Step 2 Clinical Skills Patient Note Corpus Victoria Yaneva1 Janet Mee1 Le An Ha2 Polina Harik1 Michael Jodoin1 Alex J. Mechaber1 1National Board of Medical Examiners, Philadelphia, USA {vyaneva, jmee, pharik, mjodoin, amechaber}@nbme.org 2University of Wolverhamton, UK ha.l.a@wlv.ac.uk Abstract This paper presents a corpus of 43,985 clinical patient notes (PNs) written by 35,156 exami- nees during the high-stakes USMLE® Step 2 Clinical Skills examination. In this exam, ex- aminees interact with standardized patients - people trained to portray simulated scenarios called clinical cases. For each encounter, an examinee writes a PN, which is then scored by physician raters using a rubric of clini- cal concepts, expressions of which should be present in the PN. The corpus features PNs from 10 clinical cases, as well as the clinical concepts from the case rubrics. A subset of 2,840 PNs were annotated by 10 physician experts such that all 143 concepts from the case rubrics (e.g., shortness of breath) were mapped to 34,660 PN phrases (e.g., dyspnea, difficulty breathing). The corpus is available via a data sharing agreement with NBME and can be requested at https://www.nbme.org/ services/data-sharing. 1 Introduction Large clinical text corpora are both one of the most needed and one of the least available resources in biomedical NLP, largely due to patient confiden- tiality considerations and expert annotation cost. This has been identified as a main reason for lag- ging progress in biomedical NLP compared to the general NLP domain (Chapman et al., 2011), and is evidenced by the fact that MIMIC-III (Johnson et al., 2016) is the only freely available large cor- pus of clinical notes to date (Section 2). As a re- sult, biomedical NLP is heavily reliant on corpora of PubMed scientific abstracts,1 whose academic language is in stark contrast to the often ungram- matical and telegraphic text constructions found in clinical notes. A known example of how the lack of shared clin- ical note corpora affects application development 1See BLURB: https://microsoft.github.io/BLURB/ is the task of NLP-assisted scoring of clinical pa- tient notes (PNs) written during exams. In medical education, students are often assessed through en- counters with standardized patients - people trained to portray simulated scenarios called clinical cases. For each such encounter, the student is expected to perform a history and physical examination, de- termine differential diagnoses, and then document their findings in a PN. This assessment format is ubiquitous in medical education due to the impor- tant clinical skills it measures (van der Vleuten and Swanson, 1990; Wang et al., 2021), however, there is a significant cost associated with the manual scoring of the produced PNs by expert physician raters, as well as potential for human error and bias (Engelhard Jr et al., 2018). There has been fragmented effort by individual institutions to train in-house NLP systems for clin- ical text scoring, …

Content truncated to 500 words