Can Language Models be Biomedical Knowledge Bases?

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4723–4734 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 4723 Can Language Models be Biomedical Knowledge Bases? Mujeen Sung1 Jinhyuk Lee1,2† Sean S. Yi1 Minji Jeon3 Sungdong Kim4 Jaewoo Kang1† Korea University1 Princeton University2 Icahn School of Medicine at Mount Sinai3 NAVER AI Lab4 {mujeensung,jinhyuk_lee,seanswyi,kangj}@korea.ac.kr minji.jeon@mssm.edu sungdong.kim@navercorp.com Abstract Pre-trained language models (LMs) have be- come ubiquitous in solving various natural lan- guage processing (NLP) tasks. There has been increasing interest in what knowledge these LMs contain and how we can extract that knowledge, treating LMs as knowledge bases (KBs). While there has been much work on probing LMs in the general domain, there has been little attention to whether these power- ful LMs can be used as domain-specific KBs. To this end, we create the BIOLAMA bench- mark, which is comprised of 49K biomedical factual knowledge triples for probing biomedi- cal LMs. We find that biomedical LMs with re- cently proposed probing methods can achieve up to 18.51% Acc@5 on retrieving biomedi- cal knowledge. Although this seems promis- ing given the task difficulty, our detailed analy- ses reveal that most predictions are highly cor- related with prompt templates without any sub- jects, hence producing similar results on each relation and hindering their capabilities to be used as domain-specific KBs. We hope that BIOLAMA can serve as a challenging bench- mark for biomedical factual probing.1 1 Introduction Recent success in natural language processing can be largely attributed to powerful pre-trained lan- guage models (LMs) that learn contextualized rep- resentations of words from large amounts of un- structured corpora (Peters et al., 2018; Devlin et al., 2019). There have been recent works in probing how much knowledge these LMs contain in their parameters (Petroni et al., 2019) and how to effec- tively extract such knowledge. (Shin et al., 2020; Jiang et al., 2020b; Zhong et al., 2021). 1https://github.com/dmis-lab/BioLAMA †Corresponding authors. Input Model Output (Dante, born in) (osteomyelitis, treatment) BERT Florence Insulin ciprofloxacin (a) LAMA (b) BioLAMA BERT BioBERT Figure 1: Comparison of LAMA (Petroni et al., 2019) and BIOLAMA. (a) LAMA tests general knowledge of LMs. (b) BIOLAMA tests expert-level biomedical knowledge of LMs such as a treatment for a disease. While factual probing of LMs has attracted much attention from researchers, a more practical appli- cation would be to leverage the power of domain- specific LMs (Beltagy et al., 2019; Lee et al., 2020) as domain knowledge bases (KBs). Unlike recent works that probe general domain knowledge, we ask whether it is also possible to retrieve expert knowledge from LMs. Specifically, we tune our focus on factual knowledge probing for the biomed- ical domain as shown in Figure 1. To inspect the potential utility of LMs as biomed- ical KBs, we create and release the Biomedical LAnguage Model Analysis (BIOLAMA) probe. BIOLAMA consists of 49K biomedical factual triples whose relations have been manually curated from three different knowledge sources: the Com- parative Toxicogenomics Database (CTD), the Uni- fied …

Content truncated to 500 words