Avoiding the Hypothesis-Only Bias in Natural Language Inference via Ensemble Adversarial Training

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 8281–8291, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 8281 Avoiding the Hypothesis-Only Bias in Natural Language Inference via Ensemble Adversarial Training Joe Stacey University College London ucakjd0@ucl.ac.uk Pasquale Minervini University College London p.minervini@ucl.ac.uk Haim Dubossarsky University of Cambridge hd423@cam.ac.uk Sebastian Riedel University College London s.riedel@cs.ucl.ac.uk Tim Rockt¨aschel University College London t.rocktaschel@cs.ucl.ac.uk Abstract Natural Language Inference (NLI) datasets contain annotation artefacts resulting in spu- rious correlations between the natural lan- guage utterances and their respective entail- ment classes. These artefacts are exploited by neural networks even when only considering the hypothesis and ignoring the premise, lead- ing to unwanted biases. Belinkov et al. (2019b) proposed tackling this problem via adversarial training, but this can lead to learned sentence representations that still suffer from the same biases. We show that the bias can be reduced in the sentence representations by using an en- semble of adversaries, encouraging the model to jointly decrease the accuracy of these differ- ent adversaries while fitting the data. This ap- proach produces more robust NLI models, out- performing previous de-biasing efforts when generalised to 12 other NLI datasets (Belinkov et al., 2019a; Mahabadi et al., 2020). In ad- dition, we find that the optimal number of adversarial classifiers depends on the dimen- sionality of the sentence representations, with larger sentence representations being more dif- ficult to de-bias while benefiting from using a greater number of adversaries. 1 Introduction NLI datasets are known to contain artefacts associ- ated with their human annotation processes (Guru- rangan et al., 2018). Neural models are particularly prone to picking up on artefacts, relying on these bi- ases and spurious correlations rather than acquiring a true understanding of the task. Because these arte- facts are often dataset specific (Poliak et al., 2018; Tsuchiya, 2018), models that rely on these arte- facts consequently generalise poorly when tested on other datasets (Belinkov et al., 2019a). One way to alleviate this problem is via adver- sarial training: the task classifier and an adver- sarial classifier jointly share an encoder, with the adversarial classifier trained to produce the correct predictions by analysing the artefacts in the training data. The encoder optimises the training objective while also reducing the performance of the adver- sarial classifier. In this context, adversarial training aims to produce sentence representations that do not incorporate information about the artefacts (or bias) in the data, resulting in less biased models that generalise better. Previous studies show that adversarial training is associated with better generalisation performance across other datasets, although there are concerns that the biases are not removed from the model sentence representations, with classifiers able to relearn such biases from the representations after these are frozen (Belinkov et al., 2019b; Elazar and Goldberg, 2018). It is therefore unclear whether any improvements are as a result of the de-biasing, and whether actually removing these biases from the model representations will further improve gen- eralisation. We focus our effort on this discrepancy …

Content truncated to 500 words