KILT: a Benchmark for Knowledge Intensive Language Tasks

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2523–2544 June 6–11, 2021. ©2021 Association for Computational Linguistics 2523 KILT: a Benchmark for Knowledge Intensive Language Tasks Fabio Petroni1 Aleksandra Piktus1 Angela Fan1,3 Patrick Lewis1,2 Majid Yazdani1 Nicola De Cao6 James Thorne4 Yacine Jernite5 Vladimir Karpukhin1 Jean Maillard1 Vassilis Plachouras1 Tim Rocktäschel1,2 Sebastian Riedel1,2 1Facebook AI Research 2University College London 3LORIA 4University of Cambridge 5HuggingFace 6University of Amsterdam Abstract Challenging problems such as open-domain question answering, fact checking, slot filling and entity linking require access to large, exter- nal knowledge sources. While some models do well on individual tasks, developing gen- eral models is difficult as each task might re- quire computationally expensive indexing of custom knowledge sources, in addition to ded- icated infrastructure. To catalyze research on models that condition on specific informa- tion in large textual resources, we present a benchmark for knowledge-intensive language tasks (KILT). All tasks in KILT are grounded in the same snapshot of Wikipedia, reduc- ing engineering turnaround through the re- use of components, as well as accelerating research into task-agnostic memory architec- tures. We test both task-specific and gen- eral baselines, evaluating downstream perfor- mance in addition to the ability of the mod- els to provide provenance. We find that a shared dense vector index coupled with a seq2seq model is a strong baseline, out- performing more tailor-made approaches for fact checking, open-domain question answer- ing and dialogue, and yielding competitive re- sults on entity linking and slot filling, by gen- erating disambiguated text. KILT data and code are available at https://github.com/ facebookresearch/KILT.1 1 Introduction There has been substantial progress on natural lan- guage processing tasks where the inputs are short textual contexts such as a sentences, paragraphs, or perhaps a handful of documents. Critically, we have seen the emergence of general-purpose archi- tectures and pre-trained models that can be applied to a wide range of such tasks (Devlin et al., 2019). However, for many real world problems, process- ing at this local level is insufficient. For example, 1and at https://huggingface.co/datasets? search=kilt in open-domain question answering (Chen et al., 2017) models need to find answers within a large corpus of text. Fact checking a claim (Thorne et al., 2018a) requires models to find evidence, often on the web. In knowledgeable open dialogue (Dinan et al., 2019), models need access to knowledge from large corpora to sustain informed conversa- tions. In general, solving knowledge-intensive tasks requires–even for humans–access to a large body of information. Like in Information Retrieval (IR) this involves satisfying an information need lever- aging large collections of text (Manning et al., 2008). However, while IR focuses of finding rel- evant material (usually documents), the tasks we consider focus on more fine-grained behavior, such as producing specific answers to queries. For such knowledge-intensive tasks, general infrastructure and architectures across tasks have yet to emerge, and fundamental research questions remain open. For example, while it was long assumed that non- parametric and …

Content truncated to 500 words