How Much Knowledge Can You Pack Into the Parameters of a Language Model?

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 5418–5426, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 5418 How Much Knowledge Can You Pack Into the Parameters of a Language Model? Adam Roberts∗ Google adarob@google.com Colin Raffel∗ Google craffel@gmail.com Noam Shazeer Google noam@google.com Abstract It has recently been observed that neural lan- guage models trained on unstructured text can implicitly store and retrieve knowledge using natural language queries. In this short pa- per, we measure the practical utility of this approach by fine-tuning pre-trained models to answer questions without access to any exter- nal context or knowledge. We show that this approach scales with model size and performs competitively with open-domain systems that explicitly retrieve answers from an external knowledge source when answering questions. To facilitate reproducibility and future work, we release our code and trained models.1 1 Introduction Big, deep neural language models that have been pre-trained on unlabeled text have proven to be extremely performant when fine-tuned on down- stream Natural Language Processing (NLP) tasks (Devlin et al., 2018; Yang et al., 2019; Liu et al., 2019; Lan et al., 2019; Raffel et al., 2019). In- terestingly, it has also recently been observed that these models can internalize a sort of implicit “knowledge base” after pre-training (Petroni et al., 2019; Jiang et al., 2019; Talmor et al., 2019). This behavior is potentially useful because 1) the knowledge is built up by pre-training on unstruc- tured and unlabeled text data, which is freely avail- able in huge quantities on the Internet (Raffel et al., 2019; Wenzek et al., 2019), and 2) it is pos- sible to retrieve information using informal natural language queries since these pre-trained language models excel when fine-tuned on natural language understanding tasks. ∗ Equal contribution. Noam suggested trying T5 on open-domain QA and coded and ran initial experiments on TriviaQA showing improved performance with model size. Adam wrote the code and ran most experiments. Colin set the research scope, wrote the paper, and ran a few experiments. 1https://goo.gle/t5-cbqa President Franklin <M> born <M> January 1882. Our <M> hand-picked and sun-dried <M> orchard in Georgia. Lily couldn't <M>. The waitress had brought the largest <M> of chocolate cake <M> seen. T5 D. Roosevelt was <M> in believe her eyes <M> piece <M> she had ever peaches are <M> at our When was Franklin D. Roosevelt born? T5 1882 President Franklin D. Roosevelt was born in January 1882. Pre-training Fine-tuning Figure 1: T5 is pre-trained to fill in dropped-out spans of text (denoted by <M>) from documents in a large, unstructured text corpus. We fine-tune T5 to answer questions without inputting any additional information or context. This forces T5 to answer questions based on “knowledge” that it internalized during pre-training. Past work investigating “language models as knowledge bases” has typically tried to under- stand the scope of the information stored in the model using synthetic tasks that are similar to the pre-training objective (Petroni et al., 2019; Jiang et al., 2019) and/or …

Content truncated to 500 words