Debiasing Methods in Natural Language Understanding Make Bias More Accessible

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1545–1557 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 1545 Debiasing Methods in Natural Language Understanding Make Bias More Accessible Michael Mendelson Yonatan Belinkov∗ michael.me@cs.technion.ac.il belinkov@technion.ac.il The Henry and Marilyn Taub Faculty of Computer Science Technion – Israel Institute of Technology Abstract Model robustness to bias is often determined by the generalization on carefully designed out-of-distribution datasets. Recent debias- ing methods in natural language understanding (NLU) improve performance on such datasets by pressuring models into making unbiased predictions. An underlying assumption behind such methods is that this also leads to the dis- covery of more robust features in the model’s inner representations. We propose a general probing-based framework that allows for post- hoc interpretation of biases in language mod- els, and use an information-theoretic approach to measure the extractability of certain biases from the model’s representations. We exper- iment with several NLU datasets and known biases, and show that, counter-intuitively, the more a language model is pushed towards a debiased regime, the more bias is actually en- coded in its inner representations.1 1 Introduction State of the art neural language models such as BERT (Devlin et al., 2019) usually work by pre- training an encoder to learn universal word repre- sentations, and then fine-tuning it on some classifi- cation or regression task. From a robustness point of view, such pretrain-and-fine-tune pipelines are known to be prone to biases that are present in data (Gururangan et al., 2018; Poliak et al., 2018; Mc- Coy et al., 2019; Schuster et al., 2019). Various methods were proposed to mitigate such biases in a form of robust training, where a bias model is trained to capture the bias and then used to relax the predictions of a main model, so that it can focus less on biased examples and more on the “hard”, more challenging examples (Clark et al., 2019; Ma- habadi et al., 2020; Utama et al., 2020b; Sanh et al., ∗Supported by the Viterbi Fellowship in the Center for Computer Engineering at the Technion. 1Our code and data are available at: https://github. com/technion-cs-nlp/bias-probing. 0 10 20 30 40 50 ∆ Robustness 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 ∆ Bias Extractability ρ = 0.626 Figure 1: Amount of subsequence bias extracted from different language models vs. the robustness of mod- els to the bias. Robustness is measured as improve- ment of the model on out-of-distribution examples, while extractability is measured as the improvement of the probe’s ability to extract the bias from a debiased model, compared to the baseline. 2021, inter alia). Then, the resulting model is evalu- ated on out-of-distribution (o.o.d) data, in the form of challenge datasets containing “hard” examples that were deliberately constructed to be anti-biased. Examples of such datasets include HANS (McCoy et al., 2019) for natural language inference (NLI) and FEVER-Symmetric (Schuster et al., 2019) for fact verification. An underlying assumption behind this methodology is that better generalization out of distribution also means …

Content truncated to 500 words