Large language models are few-shot clinical information extractors

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1998 - 2022 December 7-11, 2022 ©2022 Association for Computational Linguistics Large Language Models are Few-Shot Clinical Information Extractors Monica Agrawal MIT CSAIL magrawal@mit.edu Stefan Hegselmann University of Münster stefan.hegselmann@uni-muenster.de Hunter Lang MIT CSAIL hjl@mit.edu Yoon Kim MIT CSAIL yoonkim@mit.edu David Sontag MIT CSAIL dsontag@mit.edu Abstract A long-running goal of the clinical NLP com- munity is the extraction of important variables trapped in clinical notes. However, roadblocks have included dataset shift from the general domain and a lack of public clinical corpora and annotations. In this work, we show that large language models, such as InstructGPT (Ouyang et al., 2022), perform well at zero- and few-shot information extraction from clinical text despite not being trained specifically for the clinical domain. Whereas text classification and generation performance have already been studied extensively in such models, here we ad- ditionally demonstrate how to leverage them to tackle a diverse set of NLP tasks which require more structured outputs, including span identifi- cation, token-level sequence classification, and relation extraction. Further, due to the dearth of available data to evaluate these systems, we introduce new datasets for benchmarking few- shot clinical information extraction based on a manual re-annotation of the CASI dataset (Moon et al., 2014) for new tasks1. On the clinical extraction tasks we studied, the GPT-3 systems significantly outperform existing zero- and few-shot baselines. 1 Introduction Clinical text contains a large amount of valuable information that is not captured by the struc- tured data fields in electronic health records (Zweigenbaum et al., 2007; Wang et al., 2018). However, there are significant challenges to clinical information extraction. Because clinical text contains irregularities such as ambiguous jargon and nonstandard phrasal structure, most off-the-shelf NLP tools perform poorly, and clinical text annotation requires domain expertise (Zheng et al., 2011). Further, due to the sensitive nature of clinical text, public corpora are rare and restrictively licensed. As a result, clinical 1https://huggingface.co/datasets/mitclinicalml/clinical-ie Input: The patient takes coumadin 5 mg […]. Prompt: Create a list of medications. She takes 5 mg of Coumadin and Aspirin The patient takes coumadin 5 mg daily for a TIA and she has an aspirin allergy. Zero-shot prompt: Complex post-processing (resolver) of LM output [coumadin, aspirin] One-shot example + guidance: Minimal post-processing (resolver) of LM output [coumadin, aspirin] Input: Input: He is on a statin now and we think […]. Prompt: Create a list of medications. -"statin" Input: The patient takes coumadin 5 mg […]. Prompt: Create a list of medications. -"coumadin" -"aspirin" Figure 1: Illustration of our approach using a one-shot example (green) and guidance (brown) to create a more structured LM output (blue). This significantly reduces the necessary post-processing effort of a resolver (gray). NLP datasets tend to be small and splintered across institutions (Xia and Yetisgen-Yildiz, 2012). To overcome these issues, practitioners often incorporate task-specific domain knowledge and regular expressions, even in modern deep learning pipelines, but these solutions can be brittle (Luo et al., 2020; Skreta …

Content truncated to 500 words