QAInfomax: Learning Robust Question Answering System by Mutual Information Maximization

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3370–3375, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 3370 QAInfomax: Learning Robust Question Answering System by Mutual Information Maximization Yi-Ting Yeh Yun-Nung Chen Department of Computer Science and Information Engineering National Taiwan University, Taipei, Taiwan r07922064@csie.ntu.edu.tw y.v.chen@ieee.org Abstract Standard accuracy metrics indicate that mod- ern reading comprehension systems have achieved strong performance in many ques- tion answering datasets. However, the ex- tent these systems truly understand language remains unknown, and existing systems are not good at distinguishing distractor sentences, which look related but do not actually answer the question. To address this problem, we pro- pose QAInfomax as a regularizer in reading comprehension systems by maximizing mu- tual information among passages, a question, and its answer. QAInfomax helps regularize the model to not simply learn the superficial correlation for answering questions. The ex- periments show that our proposed QAInfomax achieves the state-of-the-art performance on the benchmark Adversarial-SQuAD dataset1. 1 Introduction Question answering tasks are widely used for training and testing machine comprehension and reasoning (Rajpurkar et al., 2016; Joshi et al., 2017). However, high performance in standard au- tomatic metrics has been achieved with only su- perficial understanding, as models exploit simple correlations in the data that happen to be predictive on most test examples. Jia and Liang (2017) ad- dressed this problem and proposed an adversarial version of the SQuAD dataset, which was created by adding a distractor sentence to each paragraph. The distractor sentences challenge the model ro- bustness, and the created Adversarial-SQuAD data shows the inability of a model about distinguish- ing a sentence that actually answers the question from one that merely has words in common with it, where almost all state-of-the-art machine com- prehension systems are significantly degraded on 1The source code is publicly available at https:// github.com/MiuLab/QAInfomax. adversarial examples. Lewis and Fan (2018) argued that over-fitting to superficial biases is partially caused by discrim- inative loss functions, which saturate when sim- ple correlations allow the question to be answered confidently, leaving no incentive for further learn- ing on the example. Therefore, they designed generative QA models, which use a generative loss function in question answering instead, and showed the improvement on Adversarial-SQuAD. Instead of regularizing models by generative loss functions, we propose an alternative approach named “QAInfomax” by maximizing mutual in- formation (MI) among passages, questions, and answers, aiming at helping models be not stuck with superficial biases in the data during learning. To efficiently estimate MI, QAInfomax incorpo- rates the recently proposed deep infomax (DIM) in the model (Hjelm et al., 2018), which was proved effective in learning representations for image, au- dio (Ravanelli and Bengio, 2018), and graph do- mains (Veliˇckovi´c et al., 2018). In this work, the proposed QAInfomax further extends DIM to the text domain, and encourages the question answer- ing model to generate answers carrying informa- tion that can explain not only …

Content truncated to 500 words