Infusing Disease Knowledge into BERT for Health Question Answering, Medical Inference and Disease Name Recognition

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 4604–4614, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 4604 Infusing Disease Knowledge into BERT for Health Question Answering, Medical Inference and Disease Name Recognition Yun He1, Ziwei Zhu1, Yin Zhang1, Qin Chen2, James Caverlee1 1Texas A&M University, College Station, USA 2Fudan University, Shanghai, China {yunhe, zhuziwei, zhan13679, caverlee}@tamu.edu qin_chen@fudan.edu.cn Abstract Knowledge of a disease includes information of various aspects of the disease, such as signs and symptoms, diagnosis and treatment. This disease knowledge is critical for many health- related and biomedical tasks, including con- sumer health question answering, medical lan- guage inference and disease name recognition. While pre-trained language models like BERT have shown success in capturing syntactic, se- mantic, and world knowledge from text, we find they can be further complemented by spe- cific information like knowledge of symptoms, diagnoses, treatments, and other disease as- pects. Hence, we integrate BERT with dis- ease knowledge for improving these important tasks. Specifically, we propose a new dis- ease knowledge infusion training procedure and evaluate it on a suite of BERT models in- cluding BERT, BioBERT, SciBERT, Clinical- BERT, BlueBERT, and ALBERT. Experiments over the three tasks show that these models can be enhanced in nearly all cases, demonstrat- ing the viability of disease knowledge infusion. For example, accuracy of BioBERT on con- sumer health question answering is improved from 68.29% to 72.09%, while new SOTA re- sults are observed in two datasets. We make our data and code freely available.1 1 Introduction Human disease is “a disorder of structure or function in a human that produces specific signs or symp- toms” (Oxford-English-Dictionary, 2020). Dis- ease is one of the fundamental biological enti- ties in biomedical research and consequently it is frequently searched for in the scientific literature (Islamaj Dogan et al., 2009) and on the internet (Brownstein et al., 2009). Knowledge of a disease includes information about various aspects of the disease, like the signs 1https://github.com/heyunh2015/diseaseBERT Table 1: Disease knowledge of COVID-19 is presented from three aspects: symptoms, diagnosis and treatment (based on Wikipedia). Disease Aspect Information COVID-19 symptoms Fever is the most common symptom, but highly variable in severity and presentation, with some older... COVID-19 diagnosis The standard method of testing is real-time reverse transcription poly- merase chain reaction (rRT-PCR)... COVID-19 treatment People are managed with supportive care, which may include fluid therapy, oxygen support, and supporting... and symptoms, diagnosis, and treatment (Saleem et al., 2012; Urnes et al., 2008; Du Jeong et al., 2017). As an example, Table 1 highlights several aspects for COVID-19. Specialized disease knowledge is critical for many health-related and biomedical natural language processing (NLP) tasks, including: • Consumer health question answering (Abacha et al., 2019) - the goal is to rank candidate passages for answering questions like “What is the diagnosis of COVID-19?” as shown in Figure 1a; • Medical language inference (Romanov and Shivade, 2018) - the goal is to predict if a given hypothesis (description of a patient) can …

Content truncated to 500 words