DREAM: A Challenge Data Set and Models for Dialogue-Based Reading Comprehension

DREAM: A Challenge Data Set and Models for Dialogue-Based Reading Comprehension Kai Sun♠∗ Dian Yu♥ Jianshu Chen♥ Dong Yu♥ Yejin Choi♦, ♣ Claire Cardie♠ ♠Cornell University, Ithaca, NY, USA ♥Tencent AI Lab, Bellevue, WA, USA ♦University of Washington, Seattle, WA, USA ♣Allen Institute for Artificial Intelligence, Seattle, WA, USA ks985@cornell.edu {yudian,jianshuchen,dyu}@tencent.com yejin@cs.washington.edu cardie@cs.cornell.edu Abstract We present DREAM, the first dialogue-based multiple-choice reading comprehension data set. Collected from English as a Foreign Language examinations designed by human experts to evaluate the comprehension level of Chinese learners of English, our data set contains 10,197 multiple-choice questions for 6,444 dialogues. In contrast to existing reading comprehension data sets, DREAM is the first to focus on in-depth multi-turn multi-party dialogue understanding. DREAM is likely to present significant challenges for existing reading comprehension systems: 84% of an- swers are non-extractive, 85% of questions require reasoning beyond a single sentence, and 34% of questions also involve common- sense knowledge. We apply several popular neural reading comprehension models that primarily exploit surface information within the text and find them to, at best, just barely outperform a rule- based approach. We next investigate the effects of incorporating dialogue structure and dif- ferent kinds of general world knowledge into both rule-based and (neural and non-neural) machine learning-based reading comprehen- sion models. Experimental results on the DREAM data set show the effectiveness of dia- logue structure and general world knowledge. DREAM is available at https://dataset. org/dream/. 1 Introduction Recently a significant amount of research has fo- cused on the construction of large-scale multiple- ∗This work was done when K. S. was an intern at the Tencent AI Lab, Bellevue, WA. choice (Lai et al., 2017; Khashabi et al., 2018; Ostermann et al., 2018) and extractive (Hermann et al., 2015; Hill et al., 2016; Rajpurkar et al., 2016; Trischler et al., 2017) reading comprehension data sets (Section 2). Source documents in these data sets have generally been drawn from formal written texts such as news, fiction, and Wikipedia articles, which are commonly considered well- written, accurate, and neutral. With the goal of advancing research in ma- chine reading comprehension and facilitating dia- logue understanding, we construct and present DREAM — the first multiple-choice Dialogue- based REAding comprehension exaMination data set. We collect 10,197 questions for 6,444 multi- turn multi-party dialogues from English language exams, which are carefully designed by educa- tional experts (e.g., English teachers) to assess the comprehension level of Chinese learners of English. Each question is associated with three answer options, exactly one of which is correct. (See Table 1 for an example.) DREAM covers a variety of topics and scenarios in daily life such as conversations on the street, on the phone, in a classroom or library, at the airport or the office or a shop (Section 3). Based on our analysis of DREAM, we argue that dialogue-based reading comprehension is at least as difficult as existing non-conversational counterparts. In particular, answering 34% of DREAM questions requires unspoken common- sense knowledge, for example, unspoken scene information. This might …

Content truncated to 500 words