A Self-Training Method for Machine Reading Comprehension with Soft Evidence Extraction

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3916–3927 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 3916 A Self-Training Method for Machine Reading Comprehension with Soft Evidence Extraction Yilin Niu1∗, Fangkai Jiao2∗, Mantong Zhou1 , Ting Yao3 , Jingfang Xu3 , Minlie Huang1† 1 Department of Computer Science and Technology, Institute for Artificial Intelligence, State Key Lab of Intelligent Technology and Systems, Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing 100084, China 2 School of Computer Science and Technology, Shandong University 3 Sogou Inc., Beijing, China niuyl14@tsinghua.org.cn jiaofangkai@hotmail.com zmt.keke@gmail.com {yaoting,jingfang}@sogou-inc.com aihuang@tsinghua.edu.cn Abstract Neural models have achieved great success on machine reading comprehension (MRC), many of which typically consist of two com- ponents: an evidence extractor and an answer predictor. The former seeks the most relevant information from a reference text, while the latter is to locate or generate answers from the extracted evidence. Despite the impor- tance of evidence labels for training the evi- dence extractor, they are not cheaply accessi- ble, particularly in many non-extractive MRC tasks such as YES/NO question answering and multi-choice MRC. To address this problem, we present a Self- Training method (STM), which supervises the evidence extractor with auto-generated evi- dence labels in an iterative process. At each iteration, a base MRC model is trained with golden answers and noisy evidence labels. The trained model will predict pseudo evidence labels as extra supervision in the next itera- tion. We evaluate STM on seven datasets over three MRC tasks. Experimental results demon- strate the improvement on existing MRC mod- els, and we also analyze how and why such a self-training method works in MRC. 1 Introduction Machine reading comprehension (MRC) has re- ceived increasing attention recently, which can be roughly divided into two categories: extractive and non-extractive MRC. Extractive MRC requires a model to extract an answer span to a question from reference documents, such as the tasks in SQuAD (Rajpurkar et al., 2016) and CoQA (Reddy et al., 2019). In contrast, non-extractive MRC in- fers answers based on some evidence in reference ∗Equal contribution †Corresponding author: Minlie Huang. documents, including Yes/No question answer- ing (Clark et al., 2019), multiple-choice MRC (Lai et al., 2017; Khashabi et al., 2018; Sun et al., 2019), and open domain question answering (Dhingra et al., 2017b). As shown in Table 1, evidence plays a vital role in MRC (Zhou et al., 2019; Ding et al., 2019; Min et al., 2018), and the coarse-to- fine paradigm has been widely adopted in multiple models (Choi et al., 2017; Li et al., 2018; Wang et al., 2018) where an evidence extractor first seeks the evidence from given documents and then an answer predictor infers the answer based on the evidence. However, it is challenging to learn a good evidence extractor due to the lack of evidence labels for supervision. Manually annotating the golden evidence is ex- pensive. Therefore, some recent efforts have been dedicated to improving MRC by leveraging noisy evidence …

Content truncated to 500 words