Improving the Robustness of Question Answering Systems to Question Paraphrasing

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6065–6075 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 6065 Improving the Robustness of Question Answering Systems to Question Paraphrasing Wee Chung Gan Department of Computer Science National University of Singapore gan weechung@u.nus.edu Hwee Tou Ng Department of Computer Science National University of Singapore nght@comp.nus.edu.sg Abstract Despite the advancement of question answer- ing (QA) systems and rapid improvements on held-out test sets, their generalizability is a topic of concern. We explore the robust- ness of QA models to question paraphrasing by creating two test sets consisting of para- phrased SQuAD questions. Paraphrased ques- tions from the first test set are very similar to the original questions designed to test QA models’ over-sensitivity, while questions from the second test set are paraphrased using con- text words near an incorrect answer candidate in an attempt to confuse QA models. We show that both paraphrased test sets lead to significant decrease in performance on multi- ple state-of-the-art QA models. Using a neural paraphrasing model trained to generate multi- ple paraphrased questions for a given source question and a set of paraphrase suggestions, we propose a data augmentation approach that requires no human intervention to re-train the models for improved robustness to question paraphrasing. 1 Introduction With the release of large-scale, high-quality, and increasingly challenging question answering (QA) datasets (Rajpurkar et al., 2016; Nguyen et al., 2016; Joshi et al., 2017; Reddy et al., 2018), the research community has made rapid progress on QA systems. On the popular SQuAD dataset (Rajpurkar et al., 2016), top QA models have achieved higher evaluation scores compared to hu- man. However, since the test set is typically a randomly selected subset of the whole set of data collected, and thus follows the same distribution as the training and development sets, the perfor- mance of models on the test set tends to overes- timate the models’ ability to generalize to other unseen test data. It is thus important for QA mod- els to be evaluated on other unseen test data for a Context: ... commentators had debated whether the figure could be reached as the growth in subscriber numbers elsewhere in Europe flattened. Original Question: What was happening to subscriber numbers in other areas of Europe? Prediction: flattened Paraphrased Question: What was going on with subscriber numbers in other areas of Eu- rope? Prediction: growth Context: ... According to the Second law of thermodynamics, nonconservative forces necessarily result in energy transformations within closed systems from ordered to more random conditions as entropy increases. Original Question: What is the law of ther- modynamics associated with closed system heat exchange? Prediction: Second law of thermodynamics Paraphrased Question: What is the law of thermodynamics related to closed system heat exchange? Prediction: nonconservative forces Figure 1: Examples of brittleness to paraphrasing. Both examples show an initially correct prediction turning into a wrong prediction after small changes in the question. better indication of their generalization ability. …

Content truncated to 500 words