Improving QA Generalization by Concurrent Modeling of Multiple Biases

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 839–853 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 839 Improving QA Generalization by Concurrent Modeling of Multiple Biases Mingzhu Wu, Nafise Sadat Moosavi, Andreas R¨uckl´e, Iryna Gurevych Ubiquitous Knowledge Processing Lab (UKP-TUDA) Department of Computer Science, Technische Universit¨at Darmstadt https://www.ukp.tu-darmstadt.de Abstract Existing NLP datasets contain various biases that models can easily exploit to achieve high performances on the corresponding evaluation sets. However, focusing on dataset-specific bi- ases limits their ability to learn more gener- alizable knowledge about the task from more general data patterns. In this paper, we inves- tigate the impact of debiasing methods for im- proving generalization and propose a general framework for improving the performance on both in-domain and out-of-domain datasets by concurrent modeling of multiple biases in the training data. Our framework weights each ex- ample based on the biases it contains and the strength of those biases in the training data. It then uses these weights in the training ob- jective so that the model relies less on exam- ples with high bias weights. We extensively evaluate our framework on extractive ques- tion answering with training data from vari- ous domains with multiple biases of different strengths. We perform the evaluations in two different settings, in which the model is trained on a single domain or multiple domains simul- taneously, and show its effectiveness in both settings compared to state-of-the-art debiasing methods.1 1 Introduction As a result of annotation artifacts, existing NLP datasets contain shallow patterns that correlate with target labels (Gururangan et al., 2018; McCoy et al., 2019; Schuster et al., 2019a; Le Bras et al., 2020; Jia and Liang, 2017; Das et al., 2019). Models tend to exploit these shallow patterns—which we refer to as biases in this paper– instead of learning general knowledge about solving the target task. Existing debiasing approaches weaken the im- pact of such biases by disregarding or down- 1The code and data are available at https://github.com/UKPLab/ qa-generalization-concurrent-debiasing. weighting affected training examples. They are often evaluated using adversarial or synthetic sets that contain counterexamples, in which relying on the examined bias will result in incorrect predic- tions (Belinkov et al., 2019; Clark et al., 2019; He et al., 2019; Mahabadi et al., 2020). Importantly, the majority of existing debiasing approaches only deal with a single bias. They im- prove the performance scores on a targeted adver- sarial evaluation set, while typically resulting in performance decreases on the original datasets, or on adversarial datasets that contain different types of biases (Utama et al., 2020; Nie et al., 2019; He et al., 2019). In this paper, we show that modeling multiple bi- ases is a key factor to benefit from debiasing meth- ods for improving both in-domain performance and out-of-domain generalization, and propose a new debiasing framework for concurrent modeling of multiple biases during training. A key challenge for developing a general framework that can handle multiple biases is to properly combine them when various biases’ strength is …

Content truncated to 500 words