Self-Supervised Intermediate Fine-Tuning of Biomedical Language Models for Interpreting Patient Case Descriptions

Proceedings of the 29th International Conference on Computational Linguistics, pages 1432–1441 October 12–17, 2022. 1432 Self-Supervised Intermediate Fine-Tuning of Biomedical Language Models for Interpreting Patient Case Descriptions Israa Alghanmi∗, Luis Espinosa-Anke∗♢, Steven Schockaert∗ ∗CardiffNLP, Cardiff University, UK ♢AMPLYFI, UK {alghanmiia, espinosa-ankel, schockaerts1}@cardiff.ac.uk Abstract Interpreting patient case descriptions has emerged as a challenging problem for biomedi- cal NLP, where the aim is typically to predict diagnoses, to recommended treatments, or to answer questions about cases more generally. Previous work has found that biomedical lan- guage models often lack the knowledge that is needed for such tasks. In this paper, we aim to improve their performance through a self-supervised intermediate fine-tuning strat- egy based on PubMed abstracts. Our solution builds on the observation that many of these abstracts are case reports, and thus essentially patient case descriptions. As a general strategy, we propose to fine-tune biomedical language models on the task of predicting masked medi- cal concepts from such abstracts. We find that the success of this strategy crucially depends on the selection of the medical concepts to be masked. By ensuring that these concepts are sufficiently salient, we can substantially boost the performance of biomedical language mod- els, achieving state-of-the-art results on two benchmarks. 1 Introduction Natural Language Processing (NLP) in the biomed- ical domain poses a number of particular chal- lenges. For this reason, several Language Models (LMs) that are specialised towards the biomedical domain have been proposed, including BioBERT (Lee et al., 2020), ClinicalBERT (Alsentzer et al., 2019), SciBERT (Beltagy et al., 2019a), and Pub- MedBERT (Gu et al., 2021). Recent work has focused on analysing the capabilities of such mod- els (Jin et al., 2019; Alghanmi et al., 2021; Sung et al., 2021) and enhancing them further (He et al., 2020b; Yuan et al., 2021; Zhang et al., 2021; Fei et al., 2021). Broadly speaking, biomedical LMs have proven successful in capturing the meaning of specialised terminology, but they have been far less successful in enabling medical reasoning, e.g. Question: A 38-year-old woman comes to the emer- gency department because of progressive headache, blurry vision, and nausea for 1 day. Four days ago, she was diagnosed with a right middle ear infection. She appears lethargic. Her temperature is 39.1°C (102.3°F), and blood pressure is 148/95 mm Hg. Ophthalmologic examination shows bilateral swelling of the optic disc. The corneal reflex in the right eye is absent. Sensation to touch is reduced on the upper right side of the face. Serum studies show increased concentrations of fibrin degradation products. Which of the following is the most likely diagnosis? (A) Cerebral venous thrombosis (B) Hypertensive emergency (C) Subarachnoid hemorrhage (D) Viral meningitis Table 1: Example of a question from MedQA, along with the answer candidates. for predicting a likely diagnosis from a given pa- tient case description. Currently, the main strate- gies for alleviating this latter issue have centered on incorporating structured knowledge, especially in the form of knowledge graphs. For example, Meng et al. (2021) proposed a method to integrate …

Content truncated to 500 words