Static Embeddings as Efficient Knowledge Bases?

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2353–2363 June 6–11, 2021. ©2021 Association for Computational Linguistics 2353 Static Embeddings as Efficient Knowledge Bases? Philipp Dufter∗, Nora Kassner∗, Hinrich Schütze Center for Information and Language Processing (CIS), LMU Munich, Germany {philipp,kassner}@cis.lmu.de Abstract Recent research investigates factual knowl- edge stored in large pretrained language mod- els (PLMs). Instead of structural knowledge base (KB) queries, masked sentences such as “Paris is the capital of [MASK]” are used as probes. The good performance on this analy- sis task has been interpreted as PLMs becom- ing potential repositories of factual knowledge. In experiments across ten linguistically diverse languages, we study knowledge contained in static embeddings. We show that, when re- stricting the output space to a candidate set, simple nearest neighbor matching using static embeddings performs better than PLMs. E.g., static embeddings perform 1.6% points better than BERT while just using 0.3% of energy for training. One important factor in their good comparative performance is that static embed- dings are standardly learned for a large vocab- ulary. In contrast, BERT exploits its more sophisticated, but expensive ability to com- pose meaningful representations from a much smaller subword vocabulary. 1 Introduction Pretrained language models (PLMs) (Peters et al., 2018; Howard and Ruder, 2018; Devlin et al., 2019) can be finetuned to a variety of natural language processing (NLP) tasks and then generally yield high performance. Increasingly, these models and their generative variants (e.g., GPT, Brown et al., 2020) are used to solve tasks by simple text gen- eration, without any finetuning. This motivated research on how much knowledge is contained in PLMs: Petroni et al. (2019) used models pretrained with a masked language objective to answer cloze- style templates such as: (Ex1) Paris is the capital of [MASK]. Using this methodology, Petroni et al. (2019) showed that PLMs capture some knowledge im- plicitly. This has been interpreted as suggesting ∗ Equal contribution - random order. Model Vocabulary Size p1 LAMA LAMA-UHN Oracle 22.0 23.7 BERT 30k 39.6 30.7 mBERT 110k 36.3 27.4 fastText BERT-30k 26.9 16.8 mBERT-110k 27.5 17.8 30k 16.4 5.8 120k 34.3 25.0 250k 37.7 29.0 500k 39.9 31.8 1000k 41.2 33.4 Table 1: Results for majority oracle, BERT, mBERT and fastText. Static fastText embeddings are com- petitive and outperform BERT for large vocabularies. BERT and mBERT use their subword vocabularies. For fastText, we use BERT/mBERT’s vocabularies and newly trained wordpiece vocabularies on Wikipedia. that PLMs are promising as repositories of factual knowledge. In this paper, we present evidence that simple static embeddings like fastText perform as well as PLMs in the context of answering knowl- edge base (KB) queries. Answering KB queries can be decomposed into two subproblems, typing and ranking. Typing refers to the problem of pre- dicting the correct type of the answer entity; e.g., “country” is the correct type for [MASK] in (Ex1), a task that PLMs seem to be good at. Ranking consists of finding the …

Content truncated to 500 words