Learning to Ask Unanswerable Questions for Machine Reading Comprehension

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4238–4248 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 4238 Learning to Ask Unanswerable Questions for Machine Reading Comprehension Haichao Zhu†∗, Li Dong‡, Furu Wei‡, Wenhui Wang‡, Bing Qin†♮, Ting Liu†♮ †Harbin Institute of Technology, Harbin, China ‡Microsoft Research, Beijing, China ♮Peng Cheng Laboratory, Shenzhen, China {hczhu,qinb,tliu}@ir.hit.edu.cn {lidong1,fuwei,wenwan}@microsoft.com Abstract Machine reading comprehension with unan- swerable questions is a challenging task. In this work, we propose a data augmentation technique by automatically generating relevant unanswerable questions according to an an- swerable question paired with its correspond- ing paragraph that contains the answer. We introduce a pair-to-sequence model for unan- swerable question generation, which effec- tively captures the interactions between the question and the paragraph. We also present a way to construct training data for our ques- tion generation models by leveraging the ex- isting reading comprehension dataset. Exper- imental results show that the pair-to-sequence model performs consistently better compared with the sequence-to-sequence baseline. We further use the automatically generated unan- swerable questions as a means of data aug- mentation on the SQuAD 2.0 dataset, yielding 1.9 absolute F1 improvement with BERT-base model and 1.7 absolute F1 improvement with BERT-large model. 1 Introduction Extractive reading comprehension (Hermann et al., 2015; Rajpurkar et al., 2016) obtains great attentions from both research and industry in recent years. End-to-end neural models (Seo et al., 2017; Wang et al., 2017; Yu et al., 2018) have achieved remarkable performance on the task if answers are assumed to be in the given paragraph. Nonetheless, the current systems are still not good at deciding whether no answer is presented in the context (Rajpurkar et al., 2018). For unanswerable questions, the systems are supposed to abstain from answering rather than making unreliable guesses, which is an embodiment of language understanding ability. ∗ Contribution during internship at Microsoft Research Asia. Title: Victoria (Australia) Paragraph: . . . Public schools, also known as state or government schools, are funded and run directly by the Victoria Department of Education . Students do not pay tuition fees, but some extra costs are levied. Private fee- paying schools include parish schools . . . Ans. Question: What organization runs the public schools in Victoria? UnAns. Question: What organization runs the waste management in Victoria? (Plausible) Answer: Victoria Department of Education Figure 1: An example taken from the SQuAD 2.0 dataset. The annotated (plausible) answer span in the paragraph is used as a pivot to align the pair of answer- able and unanswerable questions. We attack the problem by automatically gener- ating unanswerable questions for data augmenta- tion to improve question answering models. The generated unanswerable questions should not be too easy for the question answering model so that data augmentation can better help the model. For example, a simple baseline method is randomly choosing a question asked for another paragraph, and using it as an unanswerable question. How- ever, it would be trivial to determine whether the …

Content truncated to 500 words