Evidence Sentence Extraction for Machine Reading Comprehension

Proceedings of the 23rd Conference on Computational Natural Language Learning, pages 696–707 Hong Kong, China, November 3-4, 2019. c⃝2019 Association for Computational Linguistics 696 Evidence Sentence Extraction for Machine Reading Comprehension Hai Wang1* Dian Yu2 Kai Sun3* Jianshu Chen2 Dong Yu2 David McAllester1 Dan Roth4 1Toyota Technological Institute at Chicago, Chicago, IL, USA 2Tencent AI Lab, Bellevue, WA, USA 3Cornell, Ithaca, NY, USA 4University of Pennsylvania, Philadelphia, PA, USA {haiwang,mcallester}@ttic.edu, ks985@cornell.edu, {yudian,jianshuchen,dyu}@tencent.com, danroth@seas.upenn.edu Abstract Remarkable success has been achieved in the last few years on some limited machine read- ing comprehension (MRC) tasks. However, it is still difficult to interpret the predictions of existing MRC models. In this paper, we fo- cus on extracting evidence sentences that can explain or support the answers of multiple- choice MRC tasks, where the majority of an- swer options cannot be directly extracted from reference documents. Due to the lack of ground truth evidence sen- tence labels in most cases, we apply distant supervision to generate imperfect labels and then use them to train an evidence sentence extractor. To denoise the noisy labels, we apply a recently proposed deep probabilistic logic learning framework to incorporate both sentence-level and cross-sentence linguistic indicators for indirect supervision. We feed the extracted evidence sentences into exist- ing MRC models and evaluate the end-to-end performance on three challenging multiple- choice MRC datasets: MultiRC, RACE, and DREAM, achieving comparable or better per- formance than the same models that take as in- put the full reference document. To the best of our knowledge, this is the first work extracting evidence sentences for multiple-choice MRC. 1 Introduction Recently, there have been increased interests in machine reading comprehension (MRC). In this work, we mainly focus on multiple-choice MRC (Richardson et al., 2013; Mostafazadeh et al., 2016; Ostermann et al., 2018): given a document and a question, the task aims to select the correct answer option(s) from a small num- ber of answer options associated with this ques- * This work was done when H. W. and K. S. were at Tencent AI Lab, Bellevue, WA. tion. Compared to extractive and abstractive MRC tasks (e.g., (Rajpurkar et al., 2016; Koˇcisk`y et al., 2018; Reddy et al., 2019)) where most questions can be answered using spans from the reference documents, the majority of answer options cannot be directly extracted from the given texts. Existing multiple-choice MRC models (Wang et al., 2018b; Radford et al., 2018) take as input the entire reference document and seldom offer any explanation, making interpreting their predictions extremely difficult. It is a natural choice for hu- man readers to use sentences from a given text to explain why they select a certain answer option in reading tests (Bax, 2013). In this paper, as a pre- liminary attempt, we focus on exacting evidence sentences that entail or support a question-answer pair from the given reference document. For extractive MRC tasks, information retrieval techniques can be very strong baselines to ex- tract sentences that contain questions and their an- swers when questions provide …

Content truncated to 500 words