The NarrativeQA Reading Comprehension Challenge

The NarrativeQA Reading Comprehension Challenge Tom´aˇs Koˇcisk´y†‡ Jonathan Schwarz† Phil Blunsom†‡ Chris Dyer† Karl Moritz Hermann† G´abor Melis† Edward Grefenstette† †DeepMind ‡University of Oxford {tkocisky,schwarzjn,pblunsom,cdyer,kmh,melisgl,etg}@google.com Abstract Reading comprehension (RC)—in contrast to information retrieval—requires integrating in- formation and reasoning about events, enti- ties, and their relations across a full document. Question answering is conventionally used to assess RC ability, in both artificial agents and children learning to read. However, existing RC datasets and tasks are dominated by ques- tions that can be solved by selecting answers using superficial information (e.g., local con- text similarity or global term frequency); they thus fail to test for the essential integrative as- pect of RC. To encourage progress on deeper comprehension of language, we present a new dataset and set of tasks in which the reader must answer questions about stories by reading entire books or movie scripts. These tasks are designed so that successfully answering their questions requires understanding the underly- ing narrative rather than relying on shallow pattern matching or salience. We show that al- though humans solve the tasks easily, standard RC models struggle on the tasks presented here. We provide an analysis of the dataset and the challenges it presents. 1 Introduction Natural language understanding seeks to create mod- els that read and comprehend text. A common strat- egy for assessing the language understanding capa- bilities of comprehension models is to demonstrate that they can answer questions about documents they read, akin to how reading comprehension is tested in children when they are learning to read. After read- ing a document, a reader usually can not reproduce Title: Ghostbusters II Question: How is Oscar related to Dana? Answer: her son Summary snippet: ...Peter’s former girlfriend Dana Barrett has had a son, Oscar. .. Story snippet: DANA (setting the wheel brakes on the buggy) Thank you, Frank. I’ll get the hang of this eventually. She continues digging in her purse while Frank leans over the buggy and makes funny faces at the baby, OSCAR, a very cute nine-month old boy. FRANK (to the baby) Hiya, Oscar. What do you say, slugger? FRANK (to Dana) That’s a good-looking kid you got there, Ms. Barrett. Figure 1: Example question–answer pair. The snippets here were extracted by humans from summaries and the full text of movie scripts or books, respectively, and are not provided to the model as supervision or at test time. Instead, the model will need to read the full text and lo- cate salient snippets based solely on the question and its reading of the document in order to generate the answer. the entire text from memory, but often can answer questions about underlying narrative elements of the document: the salient entities, events, places, and the relations between them. Thus, testing understanding requires the creation of questions that examine high- level abstractions instead of just facts occurring in one sentence at a time. Unfortunately, superficial questions about a docu- ment may often be answered successfully (by both humans and machines) using a …

Content truncated to 500 words