Multi-Strategy Knowledge Distillation Based Teacher-Student Framework for Machine Reading Comprehension

Multi-Strategy Knowledge Distillation Based Teacher-Student Framework for Machine Reading Comprehension Xiaoyan Yu1,2,3∗, Qingbin Liu1,2∗, Shizhu He1,2, Kang Liu1,2, Shengping Liu5, Jun Zhao1,2, Yongbin Zhou3,4 1 School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, 100049, China 2 National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing, 100190, China 3 Institute of Information Engineering, Chinese Academy of Sciences 4 School of Cyber Security, Nanjing University of Science and Technology 5 Beijing Unisound Information Technology Co., Ltd {xiaoyan.yu, qingbin.liu, shizhu.he, kliu, jzhao}@nlpr.ia.ac.cn zhouyongbin@njust.edu.cn, liushengping@unisound.com Abstract The irrelevant information in documents poses a great challenge for machine reading compre- hension (MRC). To deal with such a challenge, current MRC models generally fall into two separate parts: evidence extraction and answer prediction, where the former extracts the key evi- dence corresponding to the question, and the latter predicts the answer based on those sentences. However, such pipeline paradigms tend to accumulate errors, i.e. extracting the incorrect evi- dence results in predicting the wrong answer. In order to address this problem, we propose a Multi-Strategy Knowledge Distillation based Teacher-Student framework (MSKDTS) for ma- chine reading comprehension. In our approach, we first take evidence and document respec- tively as the input reference information to build a teacher model and a student model. Then the multi-strategy knowledge distillation method transfers the knowledge from the teacher model to the student model at both feature and prediction level through knowledge distillation approach. Therefore, in the testing phase, the enhanced student model can predict answer similar to the teacher model without being aware of which sentence is the corresponding evidence in the docu- ment. Experimental results on the ReCO dataset demonstrate the effectiveness of our approach, and further ablation studies prove the effectiveness of both knowledge distillation strategies. 1 Introduction Machine reading comprehension (MRC) is a task that enables machines to read and understand natural language documents to answer questions. Since it well indicates the ability of machines in interpreting natural language as well as having a wide range of application scenarios, it has attracted extensive at- tention from academia and industry over the recent years. Prevailing MRC datasets define their tasks as either extracting spans from reference documents to answer questions, such as SQuAD (Rajpurkar et al., 2016) and CoQA (Reddy et al., 2019), or inferring answers based on pieces of evidence from a given document, which is also referred to as non-extractive MRC, including multiple-choice MRC (Richardson et al., 2013; Lai et al., 2017), open domain question answering (Dhingra et al., 2017) and so on. Current MRC faces the significant challenge of the irrelevant information in documents causing neg- ative impact on answer predicting. Therefore, our aim is to engage the model to focus on evidence sentences in documents and using them to answer corresponding questions accurately. To illustrate, consider the example shown in Figure 1 (adapted from the ReCO dataset (Wang et al., 2020)). In this *: Equal contribution ©2021 China National Conference on Computational Linguistics Published under Creative Commons Attribution …

Content truncated to 500 words