Improving Question Answering Model Robustness with Synthetic Adversarial Data Generation

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8830–8848 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 8830 Improving Question Answering Model Robustness with Synthetic Adversarial Data Generation Max Bartolo†∗ Tristan Thrush‡ Robin Jia‡ Sebastian Riedel†‡ Pontus Stenetorp† Douwe Kiela‡ †University College London ‡Facebook AI Research m.bartolo@cs.ucl.ac.uk Abstract Despite recent progress, state-of-the-art ques- tion answering models remain vulnerable to a variety of adversarial attacks. While dynamic adversarial data collection, in which a human annotator tries to write examples that fool a model-in-the-loop, can improve model robust- ness, this process is expensive which limits the scale of the collected data. In this work, we are the first to use synthetic adversarial data generation to make question answering mod- els more robust to human adversaries. We de- velop a data generation pipeline that selects source passages, identifies candidate answers, generates questions, then finally filters or re- labels them to improve quality. Using this ap- proach, we amplify a smaller human-written adversarial dataset to a much larger set of syn- thetic question-answer pairs. By incorporating our synthetic data, we improve the state-of-the- art on the AdversarialQA dataset by 3.7F1 and improve model generalisation on nine of the twelve MRQA datasets. We further conduct a novel human-in-the-loop evaluation and show that our models are considerably more robust to new human-written adversarial examples: crowdworkers can fool our model only 8.8% of the time on average, compared to 17.6% for a model trained without synthetic data. 1 Introduction Large-scale labelled datasets like SQuAD (Ra- jpurkar et al., 2016) and SNLI (Bowman et al., 2015) have been driving forces in natural language processing research. Over the past few years, how- ever, such “statically collected” datasets have been shown to suffer from various problems. In particu- lar, they often exhibit inadvertent spurious statisti- cal patterns that models learn to exploit, leading to poor model robustness and generalisation (Jia and Liang, 2017; Gururangan et al., 2018; Geva et al., 2019; McCoy et al., 2019; Lewis et al., 2021a). ∗ Most of this work was carried out while MB was at Facebook AI Research. Wikipedia "Old English was not static, and its usage covered a period of 700 years, from the Anglo-Saxon settlement of Britain in the 5th century to the late 11th century … Albert Baugh dates Old English from 450 to 1150, a period of full inflections, a synthetic language. Perhaps around 85 per cent …" BART <s> … settlement of Britain </s> Old English was not … </s> When did Old English begin to be used? RoBERTa 1 RoBERTa 2 RoBERTa 3 RoBERTa 4 RoBERTa 5 RoBERTa 6 5th century 450 5th century 450 5th century 5th century Q: When did Old English begin to be used? A: 5th century (i) (ii) (iii) (iv) Figure 1: The Synthetic Adversarial Data Genera- tion Pipeline showing: (i) passage selection from Wikipedia; (ii) answer candidate selection and filter- ing by model confidence (an example retained answer shown in green, and a dropped answer candidate in …

Content truncated to 500 words