Language Models as Knowledge Bases?

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 2463–2473, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 2463 Language Models as Knowledge Bases? Fabio Petroni1 Tim Rockt¨aschel1,2 Patrick Lewis1,2 Anton Bakhtin1 Yuxiang Wu1,2 Alexander H. Miller1 Sebastian Riedel1,2 1Facebook AI Research 2University College London {fabiopetroni, rockt, plewis, yolo, yuxiangwu, ahm, sriedel}@fb.com Abstract Recent progress in pretraining language mod- els on large textual corpora led to a surge of improvements for downstream NLP tasks. Whilst learning linguistic knowledge, these models may also be storing relational knowl- edge present in the training data, and may be able to answer queries structured as “fill- in-the-blank” cloze statements. Language models have many advantages over structured knowledge bases: they require no schema en- gineering, allow practitioners to query about an open class of relations, are easy to extend to more data, and require no human supervision to train. We present an in-depth analysis of the relational knowledge already present (without fine-tuning) in a wide range of state-of-the- art pretrained language models. We find that (i) without fine-tuning, BERT contains rela- tional knowledge competitive with traditional NLP methods that have some access to ora- cle knowledge, (ii) BERT also does remark- ably well on open-domain question answer- ing against a supervised baseline, and (iii) cer- tain types of factual knowledge are learned much more readily than others by standard lan- guage model pretraining approaches. The sur- prisingly strong ability of these models to re- call factual knowledge without any fine-tuning demonstrates their potential as unsupervised open-domain QA systems. The code to re- produce our analysis is available at https: //github.com/facebookresearch/LAMA. 1 Introduction Recently, pretrained high-capacity language mod- els such as ELMo (Peters et al., 2018a) and BERT (Devlin et al., 2018a) have become increasingly important in NLP. They are optimised to either predict the next word in a sequence or some masked word anywhere in a given sequence (e.g. “Dante was born in [Mask] in the year 1265.”). The parameters of these models appear to store Memory Query Answer Symbolic KB Memory Access Neural LM Memory Access (Dante, born-in, X) “Dante was born in [Mask].” Dante Florence born-in Florence Florence KB LM e.g. ELMo/BERT Figure 1: Querying knowledge bases (KB) and lan- guage models (LM) for factual knowledge. vast amounts of linguistic knowledge (Peters et al., 2018b; Goldberg, 2019; Tenney et al., 2019) use- ful for downstream tasks. This knowledge is usually accessed either by conditioning on latent context representations produced by the original model or by using the original model weights to initialize a task-specific model which is then fur- ther fine-tuned. This type of knowledge transfer is crucial for current state-of-the-art results on a wide range of tasks. In contrast, knowledge bases are effective so- lutions for accessing annotated gold-standard re- lational data by enabling queries such as (Dante, born-in, X). However, in practice we often need to extract relational data from text or other modal- …

Content truncated to 500 words