The LAMBADA dataset: Word prediction requiring a broad discourse context

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1525–1534, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics The LAMBADA dataset: Word prediction requiring a broad discourse context∗ Denis Paperno, Germ´an Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham†, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, Raquel Fern´andez‡ CIMeC - Center for Mind/Brain Sciences, University of Trento {firstname.lastname}@unitn.it, †quanpn90@gmail.com ‡Institute for Logic, Language & Computation, University of Amsterdam raquel.fernandez@uva.nl Abstract We introduce LAMBADA, a dataset to evaluate the capabilities of computational models for text understanding by means of a word prediction task. LAMBADA is a collection of narrative passages shar- ing the characteristic that human subjects are able to guess their last word if they are exposed to the whole passage, but not if they only see the last sentence preced- ing the target word. To succeed on LAM- BADA, computational models cannot sim- ply rely on local context, but must be able to keep track of information in the broader discourse. We show that LAMBADA ex- emplifies a wide range of linguistic phe- nomena, and that none of several state-of- the-art language models reaches accuracy above 1% on this novel benchmark. We thus propose LAMBADA as a challenging test set, meant to encourage the develop- ment of new models capable of genuine understanding of broad context in natural language text. 1 Introduction The recent spurt of powerful end-to-end-trained neural networks for Natural Language Processing (Hermann et al., 2015; Rockt¨aschel et al., 2016; Weston et al., 2015, a.o.) has sparked interest in tasks to measure the progress they are bringing about in genuine language understanding. Spe- cial care must be taken in evaluating such systems, since their effectiveness at picking statistical gen- eralizations from large corpora can lead to the il- lusion that they are reaching a deeper degree of understanding than they really are. For example, ∗Denis and Germ´an share first authorship. Marco, Gemma, and Raquel share senior authorship. the end-to-end system of Vinyals and Le (2015), trained on large conversational datasets, produces dialogues such as the following: (1) Human: what is your job? Machine: i’m a lawyer Human: what do you do? Machine: i’m a doctor Separately, the system responses are appropriate for the respective questions. However, when taken together, they are incoherent. haviour is somewhat parrot-like.The system be- It can locally produce perfectly sensible language fragments, but it fails to take the meaning of the broader dis- course context into account. Much research ef- fort has consequently focused on designing sys- tems able to keep information from the broader context into memory, and possibly even perform simple forms of reasoning about it (Hermann et al., 2015; Hochreiter and Schmidhuber, 1997; Ji et al., 2015; Mikolov et al., 2015; Sordoni et al., 2015; Sukhbaatar et al., 2015; Wang and Cho, 2015, a.o.). In this paper, we introduce the LAMBADA dataset (LAnguage Modeling Broadened Account for Discourse Aspects). LAMBADA pro-to poses a word prediction task where the target item is difficult to guess (for …

Content truncated to 500 words