Incorporating medical knowledge in BERT for clinical relation extraction

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5357–5366 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 5357 Incorporating medical knowledge in BERT for clinical relation extraction Arpita Roy and Shimei Pan Department of Information Systems University of Maryland, Baltimore County Maryland, USA {arpita2, shimei}@umbc.edu Abstract In recent years pre-trained language models (PLM) such as BERT have proven to be very effective in diverse NLP tasks such as Informa- tion Extraction, Sentiment Analysis and Ques- tion/Answering. Trained with massive general- domain text, these pre-trained language mod- els capture rich syntactic, semantic and dis- course information in the text. However, due to the differences between general and spe- cific domain text (e.g., Wikipedia text versus clinic notes), these models may not be ideal for domain-specific tasks (e.g., extracting clinical relations). Furthermore, it may require addi- tional medical knowledge to understand clini- cal text properly. To solve these issues, in this research, we conduct a comprehensive exami- nation of different techniques to add medical knowledge into a pre-trained BERT model for clinical relation extraction. Our best model outperformed the state-of-the-art systems on the benchmark i2b2/VA 2010 clinical relation extraction dataset. 1 Introduction In recent years pre-trained language models (PLMs) such as ELMo (Peters et al., 2017), BERT (Devlin et al., 2019), XLNet (Yang et al., 2019), and GPT (Radford et al., 2018) have be- come very popular as they can effectively boost the performance of diverse NLP tasks such as In- formation Extraction (Shi and Lin, 2019; Jia et al., 2020), Sentiment Analysis(Gao et al., 2019), Ques- tion/Answering(Lv et al., 2020) and language en- tailment (Devlin et al., 2019). These models are trained on large text corpora using self-supervised tasks such as masked language modeling (MLM) and next sentence prediction. As these models can learn meaningful context-sensitive text embed- dings, they are frequently used to encode input text in many downstream text analysis tasks. How- ever, PLMs trained on general-domain text (e.g., books, Wikipedia and webdata) may not be ideal for domain-specific NLP applications (e.g., bio- medical NLP). In this research, we explore how medical knowledge can be added to PLMs to facili- tate clinical relation extraction. Previously, significant effort has been made on adding domain knowledge into PLMs. Based on the types of knowledge added, we can group the work into two categories: integrating domain text in PLMs (Lee et al., 2020; Peng et al., 2019a; Gu et al., 2020) and integrating domain-specific knowledge graphs into PLMs (Wang et al., 2021; Zhang et al., 2019; Peters et al., 2019). In this study, we conduct a comprehensive investigation of these methods. We test their effectiveness in integrating knowledge from Unified Medical Language System (UMLS) into BERT for clinical relation extraction. Here we focus on UMLS because it is one of the most widely used bio-medical knowledge sources for clinical NLP. Among all the PLMs, we focus on BERT since it often achieves the state of the art performance on diverse NLP tasks. The main con- tributions of …

Content truncated to 500 words