Language Models as Knowledge Bases: On Entity Representations, Storage Capacity, and Paraphrased Queries

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 1772–1791 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 1772 Language Models as Knowledge Bases: On Entity Representations, Storage Capacity, and Paraphrased Queries Benjamin Heinzerling1, 2 and Kentaro Inui2, 1 1RIKEN AIP & 2Tohoku University benjamin.heinzerling@riken.jp | inui@tohoku.ac.jp Abstract Pretrained language models have been sug- gested as a possible alternative or comple- ment to structured knowledge bases. However, this emerging LM-as-KB paradigm has so far only been considered in a very limited setting, which only allows handling 21k entities whose name is found in common LM vocabularies. Furthermore, a major benefit of this paradigm, i.e., querying the KB using natural language paraphrases, is underexplored. Here we formu- late two basic requirements for treating LMs as KBs: (i) the ability to store a large num- ber facts involving a large number of entities and (ii) the ability to query stored facts. We explore three entity representations that allow LMs to handle millions of entities and present a detailed case study on paraphrased querying of facts stored in LMs, thereby providing a proof-of-concept that language models can in- deed serve as knowledge bases. 1 Introduction Language models (LMs) appear to memorize world knowledge facts during training. For example, BERT (Devlin et al., 2019) correctly answers the query “Paris is the capital of [MASK]” with “France”. This observation prompted Petroni et al. (2019) to ask if LMs can serve as an alternative or complement to structured knowledge bases (KBs), thereby introducing the idea of treating LMs as KBs: During training, the LM encounters world knowledge facts expressed in its training data, some of which are stored in some form in the LM’s pa- rameters. After training, some of the stored facts can be recovered from the LM’s parameters by means of a suitable natural language query (Fig. 1). A LM with such a “built-in” KB is useful for knowledge-intensive tasks (Petroni et al., 2020) and question answering (Roberts et al., 2020), and could improve natural language interfaces to struc- tured data (Hendrix et al., 1978; Herzig et al., 2020). Figure 1: The LM-as-KB paradigm, introduced by Petroni et al. (2019). A LM memorizes factual state- ments, which can be queried in natural language. However, this emerging LM-as-KB paradigm is faced with several foundational questions. First question: KBs contain millions of entities, while LM vocabulary size usually does not exceed 100k entries. How can millions of entities be rep- resented in LMs? Petroni et al. (2019) circum- vent this problem by only considering 21k enti- ties whose canonical name corresponds to a single token in the LM’s vocabulary, e.g., entities like “France” or “Bert”, but not “United Kingdom” or “Sesame Street”. Hence, this approach cannot han- dle entities not contained in the vocabulary and a query like “Bert is a character on [MASK]” is not answerable in this simplified setting. To answer this first question, we compare three methods for scaling LM-as-KB to millions of entities: …

Content truncated to 500 words