End-to-End Self-Debiasing Framework for Robust NLU Training

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1923–1929 August 1–6, 2021. ©2021 Association for Computational Linguistics 1923 End-to-End Self-Debiasing Framework for Robust NLU Training Abbas Ghaddar, Philippe Langlais†, Mehdi Rezagholizadeh and Ahmad Rashid Huawei Noah’s Ark Lab, Montreal Research Center, Canada †RALI-DIRO, Universit´e de Montr´eal, Montr´eal, Canada abbas.ghaddar@huawei.com, felipe@iro.umontreal.ca mehdi.rezagholizadeh@huawei.com, ahmad.rashid@huawei.com Abstract Existing Natural Language Understanding (NLU) models have been shown to incorporate dataset biases leading to strong performance on in-distribution (ID) test sets but poor per- formance on out-of-distribution (OOD) ones. We introduce a simple yet effective debias- ing framework whereby the shallow represen- tations of the main model are used to derive a bias model and both models are trained si- multaneously. We demonstrate on three well studied NLU tasks that despite its simplicity, our method leads to competitive OOD results. It significantly outperforms other debiasing ap- proaches on two tasks, while still delivering high in-distribution performance. 1 Introduction Researchers have increasingly raised concerns about the tendency of recent NLU models (De- vlin et al., 2019; Liu et al., 2019) to quickly lever- age spurious surface lexical-syntactic features (Po- liak et al., 2018; Gururangan et al., 2018; Das- gupta et al., 2018; Ghaddar and Langlais, 2017). These superficial properties, also referred as dataset biases (Shah et al., 2020; Utama et al., 2020b; Moosavi et al., 2020), result in significant perfor- mance drop on out-of-distribution (OOD) sets con- taining counterexamples to biases in the training data (McCoy et al., 2019; Schuster et al., 2019; Zhang et al., 2019; Ghaddar et al., 2021). The most common approach to tackle the prob- lem consists in training a bias model with hand- crafted features with the goal of identifying biased training examples. This information is used in a later stage to discourage the main model from adopting the naive strategy of the bias model. Sev- eral debiasing training paradigms have been pro- posed to adjust the importance of biased training samples, such as product of experts (Clark et al., 2019; He et al., 2019), learned-mixin (Clark et al., 2019), example reweighting (Schuster et al., 2019), debiased focal loss (Mahabadi et al., 2020), and confidence regularization (Utama et al., 2020a). Recently, there has been a number of endeav- ours to produce a bias model without prior knowl- edge on the targeted biases or without the need for manually designing features. Utama et al. (2020b) propose to use instead a model trained on a tiny fraction (< 1%) of the training data for few epochs as a bias model; while Clark et al. (2020) and Sanh et al. (2020) trained a low capacity model on the full training set. These approaches target the train- ing of the bias model alone, which is subsequently queried while training the main model of interest. In this paper, we propose an end-to-end debi- asing framework which does not require an extra training stage, or manual bias features engineering. The bias model is indeed a simple attention-based classification layer on top of the main model’s inter- …

Content truncated to 500 words