On Adversarial Removal of Hypothesis-only Bias in Natural Language Inference

Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM), pages 256–262 Minneapolis, June 6–7, 2019. c⃝2019 Association for Computational Linguistics 256 On Adversarial Removal of Hypothesis-only Bias in Natural Language Inference Yonatan Belinkov13∗ Adam Poliak2∗ Stuart M. Shieber1 Benjamin Van Durme2 Alexander Rush1 1Harvard University 2Johns Hopkins University 3Massachusetts Institute of Technology {belinkov,shieber,srush}@seas.harvard.edu {azpoliak,vandurme}@cs.jhu.edu Abstract Popular Natural Language Inference (NLI) datasets have been shown to be tainted by hypothesis-only biases. Adversarial learning may help models ignore sensitive biases and spurious correlations in data. We evaluate whether adversarial learning can be used in NLI to encourage models to learn representa- tions free of hypothesis-only biases. Our anal- yses indicate that the representations learned via adversarial learning may be less biased, with only small drops in NLI accuracy. 1 Introduction Popular datasets for Natural Language Inference (NLI) - the task of determining whether one sen- tence (premise) likely entails another (hypothesis) - contain hypothesis-only biases that allow mod- els to perform the task surprisingly well by only considering hypotheses while ignoring the corre- sponding premises. For instance, such a method correctly predicted the examples in Table 1 as con- tradictions. As datasets may always contain bi- ases, it is important to analyze whether, and to what extent, models are immune to or rely on known biases. Furthermore, it is important to build models that can overcome these biases. Recent work in NLP aims to build more ro- bust systems using adversarial methods (Alzantot et al., 2018; Chen & Cardie, 2018; Belinkov & Bisk, 2018, i.a.). In particular, Elazar & Gold- berg (2018) attempted to use adversarial training to remove demographic attributes from text data, with limited success. Inspired by this line of work, we use adversarial learning to add small compo- nents to an existing and popular NLI system that has been used to learn general sentence represen- tations (Conneau et al., 2017). The adversarial ∗ Equal contribution A dog runs through the woods near a cottage ▶ The dog is sleeping on the ground A person writing something on a newspaper ▶ A person is driving a fire truck A man is doing tricks on a skateboard ▶ Nobody is doing tricks Table 1: Examples from SNLI’s development set that Poliak et al. (2018)’s hypothesis-only model correctly predicted as contradictions. The first line in each sec- tion is a premise and lines with ▶ are corresponding hypotheses. The italicized words are correlated with the “contradiction” label in SNLI techniques include (1) using an external adversar- ial classifier conditioned on hypotheses alone, and (2) creating noisy, perturbed training examples. In our analyses we ask whether hidden, hypothesis- only biases are no longer present in the resulting sentence representations after adversarial learning. The goal is to build models with less bias, ideally while limiting the inevitable degradation in task performance. Our results suggest that progress on this goal may depend on which adversarial learn- ing techniques are used. Although recent work has applied adversarial learning to NLI (Minervini & Riedel, …

Content truncated to 500 words