The Impacts of Unanswerable Questions on the Robustness of Machine Reading Comprehension Models

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1543–1557 May 2-6, 2023 ©2023 Association for Computational Linguistics The Impacts of Unanswerable Questions on the Robustness of Machine Reading Comprehension Models Son Quoc Tran†,§, Phong Nguyen-Thuan Do§, Uyen Le†, Matt Kretchmar † †Denison University, Granville, OH, USA {tran_s2, le_u1, kretchmar}@denison.edu §The UIT NLP Group, Vietnam National University, Ho Chi Minh City phongdntvn@gmail.com Abstract Pretrained language models have achieved super-human performances on many Machine Reading Comprehension (MRC) benchmarks. Nevertheless, their relative inability to defend against adversarial attacks has spurred skepti- cism about their natural language understand- ing. In this paper, we ask whether training with unanswerable questions in SQuAD 2.0 can help improve the robustness of MRC mod- els against adversarial attacks. To explore that question, we fine-tune three state-of-the- art language models on either SQuAD 1.1 or SQuAD 2.0 and then evaluate their robustness under adversarial attacks. Our experiments re- veal that current models fine-tuned on SQuAD 2.0 do not initially appear to be any more ro- bust than ones fine-tuned on SQuAD 1.1, yet they reveal a measure of hidden robustness that can be leveraged to realize actual performance gains. Furthermore, we find that the robust- ness of models fine-tuned on SQuAD 2.0 ex- tends to additional out-of-domain datasets. Fi- nally, we introduce a new adversarial attack to reveal artifacts of SQuAD 2.0 that current MRC models are learning. 1 1 Introduction Machine Reading Comprehension (MRC) is a fun- damental and challenging subfield of Natural Lan- guage Processing (NLP) in which the computer simulates a human question-and-answer mecha- nism by extracting the answers to given questions based on provided contexts. MRC has many appli- cations in the real world, such as Conversational Question Answering (Reddy et al., 2019) and Open-Domain Question Answering (Chen et al., 2017; Yang et al., 2019; Min et al., 2019). With the development of recent deep learn- ing models, MRC has made significant perfor- mance gains. Many high-quality MRC datasets 1Our code is publicly available at: https://github. com/sonqt/unanswerable-robustness. Figure 1: Example of predictions to an answerable question of RoBERTa fine-tuned on SQuAD 1.1 (Ra- jpurkar et al., 2016) (v1) versus its counterpart fine- tuned on SQuAD 2.0 (Rajpurkar et al., 2018) (v2) un- der adversarial attack. While RoBERTa v1 predicts “DartFord” as the answer under attack, RoBERTa v2 knows that “DartFord” is not the correct answer but fails to focus back on “Nevada”, the correct answer for the given question. RoBERTa v2 then predicts the tested question as unanswerable. and benchmarks (Kwiatkowski et al., 2019; Joshi et al., 2017; Yang et al., 2018; Rajpurkar et al., 2018) have been proposed over the last few years. During the same time period, MRC sys- tems have also achieved many new state-of-the- art (SOTA) performances, matching or exceed- ing human-level standards on many benchmarks. Nevertheless, skepticism persists about the real ability of MRC SOTA models (Sen and Saffari, 2020; Jia and Liang, 2017; Sugawara et al., 2018, 2020). The use of …

Content truncated to 500 words