Evaluating Data Augmentation for Medication Identification in Clinical Notes

Proceedings of Recent Advances in Natural Language Processing, pages 578–585 Varna, Sep 4–6, 2023 https://doi.org/10.26615/978-954-452-092-2_063 578 Evaluating Data Augmentation for Medication Identification in Clinical Notes Jordan Koontz Ixa UPV/EHU Donostia, Basque Country, Spain jkoontz001@ikasle.ehu.eus Maite Oronoz and Alicia P´erez HiTZ - Ixa UPV/EHU Donostia, Basque Country, Spain maite.oronoz@ehu.eus alicia.perez@ehu.eus Abstract We evaluate the effectiveness of using data aug- mentation to improve the generalizability of a Named Entity Recognition model for the task of medication identification in clinical notes. We compare disparate data augmen- tation methods, namely mention-replacement and a generative model, for creating synthetic training examples. Through experiments on the n2c2 2022 Track 1 Contextualized Medication Event Extraction data set, we show that data augmentation with supplemental examples cre- ated with GPT-3 can boost the performance of a transformer-based model for small training sets. 1 Introduction Natural Language Processing (NLP) is an active area of research in healthcare, especially due to the proliferation of Electronic Health Records (EHR). EHRs contain extensive information about individ- ual patients, such as diagnoses with their corre- sponding International Classification of Disease (ICD) codes, treatment records and test results. While some medication information can be ex- tracted from the structured data in the EHRs, a substantial amount of the medication information resides in text-based narrative clinical notes (Sohn et al., 2014). The information contained in clinical notes can be useful for pharmacovigilance, com- parative effectiveness studies, and adverse event detection (Uzuner et al., 2010). The objective of the n2c2 2022 Track 1 Contextualized Medication Event Extraction was to capture multi-dimensional context of medication changes documented in clin- ical notes. The track was comprised of three sub- tasks: • Task 1: [NER] Medication Extraction • Task 2: [Event] Event Classification • Task 3: [Context] Context Classification A prerequisite for understanding medication changes in clinical documents is to successfully identify all mentions of medication in the docu- ments. However, in the clinical domain, a common challenge for training machine learning models is a lack of annotated training data. Annotating clini- cal notes can be an expensive and lengthy process that requires medical domain experts. In this paper, we set out to evaluate disparate data augmentation techniques to create supplemental training exam- ples with the hope of reducing a dependence on manual annotations while also boosting the perfor- mance of a medication identification model. First, we detail our model architecture com- prised of a transformer-based language model and a Conditional Random Fields (CRF) (Lafferty et al., 2001) component for identifying mentions of med- ication in clinical documents that obtained com- petitive results on the n2c2 2022 [NER] Medica- tion Extraction subtask. Next, we detail our data augmentation methodology for creating synthetic training examples. Finally, we evaluate the effec- tiveness of using data augmentation for the task of medication extraction in clinical documents. More- over, we evaluate the effectiveness of using data augmentation for low-resource medication extrac- tion, i.e. a scenario in which the size of a training set is small. 2 Background Early systems for medication identification re- …

Content truncated to 500 words