Looking Beyond the Surface: A Challenge Set for Reading Comprehension over Multiple Sentences

Proceedings of NAACL-HLT 2018, pages 252–262 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Looking Beyond the Surface: A Challenge Set for Reading Comprehension over Multiple Sentences Daniel Khashabi1, Snigdha Chaturvedi2, Michael Roth3, Shyam Upadhyay1, Dan Roth1 1University of Pennsylvania, 2University of California, Santa Cruz, 3Saarland University {danielkh,shyamupa,danroth}@cis.upenn.edu, snigdha@ucsc.edu, mroth@coli.uni-sb.de Abstract We present a reading comprehension chal- lenge in which questions can only be answered by taking into account information from mul- tiple sentences. We solicit and verify ques- tions and answers for this challenge through a 4-step crowdsourcing experiment. Our chal- lenge dataset contains ∼6k questions for +800 paragraphs across 7 different domains (ele- mentary school science, news, travel guides, fiction stories, etc) bringing in linguistic diver- sity to the texts and to the questions wordings. On a subset of our dataset, we found human solvers to achieve an F1-score of 86.4%. We analyze a range of baselines, including a re- cent state-of-art reading comprehension sys- tem, and demonstrate the difficulty of this challenge, despite a high human performance. The dataset is the first to study multi-sentence inference at scale, with an open-ended set of question types that requires reasoning skills. 1 Introduction Machine Comprehension of natural language text is a fundamental challenge in AI and it has re- ceived significant attention throughout the his- tory of AI (Greene, 1959; McCarthy, 1976; Re- iter, 1976; Winograd, 1980). In particular, in natural language processing (NLP) it has been studied under various settings, such as multiple- choice Question-Answering (QA) (Green Jr. et al., 1961), Reading Comprehension (RC) (Hirschman et al., 1999), Recognizing Textual Entailment (RTE) (Dagan et al., 2013) etc. The area has seen rapidly increasing interest, thanks to the existence of sizable datasets and standard benchmarks. CNN/Daily Mail (Hermann et al., 2015), SQuAD (Rajpurkar et al., 2016) and NewsQA (Trischler et al., 2016) to name a few, are some of the datasets that were released re- cently with the goal of facilitating research in ma- chine comprehension. Despite all the excitement fueled by that large data sets and the ability to directly train statistical learning models, current QA systems do not have capabilities comparable to elementary school or younger children (Clark and Etzioni, 2016). For many of these datasets, researchers point out that models neither need to ‘comprehend’ in order to correctly predict an answer, nor do they learn to ‘reason’ in a way that generalizes across datasets. For example, Khashabi et al. (2016) showed that adversarial per- turbation in candidate answers results in a signifi- cant drop in performance of a few state-of-art sci- ence QA systems. Similarly, Jia and Liang (2017) show that adding an adversarially selected sen- tence to the instances in the SQuAD datasets dras- tically reduces the performance of many of the ex- isting baselines. Chen et al. (2016) show that in the CNN/Daily Mail datasets, “the required reasoning and inference level ...is quite simple” and that a relatively simple algorithm can get almost close to the upper-bound. We believe …

Content truncated to 500 words