Probing Pre-Trained Language Models for Disease Knowledge

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3023–3033 August 1–6, 2021. ©2021 Association for Computational Linguistics 3023 Probing Pre-Trained Language Models for Disease Knowledge Israa Alghanmi, Luis Espinosa-Anke, Steven Schockaert Cardiff University, United Kingdom {alghanmiia,espinosa-ankel,schockaerts1}@cardiff.ac.uk Abstract Pre-trained language models such as Clini- calBERT have achieved impressive results on tasks such as medical Natural Language In- ference. At first glance, this may suggest that these models are able to perform medi- cal reasoning tasks, such as mapping symp- toms to diseases. However, we find that stan- dard benchmarks such as MedNLI contain rel- atively few examples that require such forms of reasoning. To better understand the medi- cal reasoning capabilities of existing language models, in this paper we introduce DisKnE, a new benchmark for Disease Knowledge Eval- uation. To construct this benchmark, we an- notated each positive MedNLI example with the types of medical reasoning that are needed. We then created negative examples by corrupt- ing these positive examples in an adversar- ial way. Furthermore, we define training-test splits per disease, ensuring that no knowledge about test diseases can be learned from the training data, and we canonicalize the formu- lation of the hypotheses to avoid the presence of artefacts. This leads to a number of binary classification problems, one for each type of reasoning and each disease. When analysing pre-trained models for the clinical/biomedical domain on the proposed benchmark, we find that their performance drops considerably. 1 Introduction Pre-trained language models (LMs) such as BERT (Devlin et al., 2019) are currently the de-facto ar- chitecture for solving most NLP tasks, and their prevalence in general language understanding tasks is today indisputable (Wang et al., 2018, 2019). Beyond generic benchmarks, it has been shown that LMs are also extremely powerful in domain- specific NLP tasks, e.g., in the biomedical do- main (Lewis et al., 2020). While there are sev- eral reasons why they are preferred over standard neural architectures, one important (and perhaps less obvious) reason is that LMs capture a sub- stantial amount of world knowledge. For instance, several authors have found that LMs are able to answer questions without having access to exter- nal resources (Petroni et al., 2019; Roberts et al., 2020), or that they exhibit commonsense knowl- edge (Forbes et al., 2019; Davison et al., 2019). To analyze the capabilities of LMs in a more system- atic way, there is a growing interest in designing probing tasks, which are now common across the NLP landscape, e.g., for word and sentence-level semantics (Paperno et al., 2016; Conneau et al., 2018). In this paper we focus on (generic and specialized) LMs in the biomedical domain, and ask the following question: what kinds of medi- cal knowledge do pre-trained LMs capture? More specifically, we focus on disease knowledge, which encompasses for instance the ability to link symp- toms to diseases, or treatments to diseases. Among the several biomedical LMs (i.e. LMs that have been pre-trained on biomedical text cor- pora) that exist today, some of the most …

Content truncated to 500 words