A Thorough Examination of the CNN/Daily Mail Reading Comprehension Task

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 2358–2367, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics A Thorough Examination of the CNN/Daily Mail Reading Comprehension Task Danqi Chen and Jason Bolton and Christopher D. Manning Computer Science Stanford University Stanford, CA 94305-9020, USA {danqi,jebolton,manning}@cs.stanford.edu Abstract Enabling a computer to understand a docu- ment so that it can answer comprehension questions is a central, yet unsolved goal of NLP. A key factor impeding its solu- tion by machine learned systems is the lim- ited availability of human-annotated data. Hermann et al. (2015) seek to solve this problem by creating over a million training examples by pairing CNN and Daily Mail news articles with their summarized bullet points, and show that a neural network can then be trained to give good performance on this task. In this paper, we conduct a thorough examination of this new reading comprehension task. Our primary aim is to understand what depth of language un- derstanding is required to do well on this task. We approach this from one side by doing a careful hand-analysis of a small subset of the problems and from the other by showing that simple, carefully designed systems can obtain accuracies of 72.4% and 75.8% on these two datasets, exceeding cur- rent state-of-the-art results by over 5% and approaching what we believe is the ceiling for performance on this task.1 1 Introduction Reading comprehension (RC) is the ability to read text, process it, and understand its meaning.2 How to endow computers with this capacity has been an elusive challenge and a long-standing goal of Arti- ficial Intelligence (e.g., (Norvig, 1978)). Genuine reading comprehension involves interpretation of 1Our code is available at https://github.com/danqi/ rc-cnn-dailymail. 2https://en.wikipedia.org/wiki/Reading_ comprehension the text and making complex inferences. Human reading comprehension is often tested by asking questions that require interpretive understanding of a passage, and the same approach has been suggested for testing computers (Burges, 2013). In recent years, there have been several strands of work which attempt to collect human-labeled data for this task – in the form of document, question and answer triples – and to learn machine learning mod- els directly from it (Richardson et al., 2013; Berant et al., 2014; Wang et al., 2015). However, these datasets consist of only hundreds of documents, as the labeled examples usually require considerable expertise and neat design, making the annotation process quite expensive. The subsequent scarcity of labeled examples prevents us from training power- ful statistical models, such as deep learning models, and would seem to prevent a system from learning complex textual reasoning capacities. Recently, researchers at DeepMind (Hermann et al., 2015) had the appealing, original idea of exploiting the fact that the abundant news articles of CNN and Daily Mail are accompanied by bullet point summaries in order to heuristically create large-scale supervised training data for the reading comprehension task. Figure 1 gives an example. Their idea is that a bullet point usually summarizes one or several aspects …

Content truncated to 500 words