HypoNLI: Exploring the Artificial Patterns of Hypothesis-only Bias in Natural Language Inference

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 6852–6860 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 6852 HypoNLI: Exploring the Artificial Patterns of Hypothesis-only Bias in Natural Language Inference Tianyu Liu1, Xin Zheng3, Baobao Chang1,2, Zhifang Sui1,2 1MOE Key Lab of Computational Linguistics, School of EECS, Peking University 2Peng Cheng Laboratory, Shenzhen, China 3Beijing University of Posts and Telecommunications tianyu0421@pku.edu.cn, zheng xin@bupt.edu.cn, chbb@pku.edu.cn, szf@pku.edu.cn Abstract Many recent studies have shown that for models trained on datasets for natural language inference (NLI), it is possible to make correct predictions by merely looking at the hypothesis while completely ignoring the premise. In this work, we manage to derive adversarial examples in terms of the hypothesis-only bias and explore eligible ways to mitigate such bias. Specifically, we extract various phrases from the hypotheses (artificial patterns) in the training sets, and show that they have been strong indicators to the specific labels. We then figure out ‘hard’ and ‘easy’ instances from the original test sets whose labels are opposite to or consistent with those indications. We also set up baselines including both pretrained models (BERT, RoBERTa, XLNet) and competitive non-pretrained models (InferSent, DAM, ESIM). Apart from the benchmark and baselines, we also investigate two debiasing approaches which exploit the artificial pattern modeling to mitigate such hypothesis-only bias: down-sampling and adversarial training. We believe those methods can be treated as competitive baselines in NLI debiasing tasks. Keywords: Natural Language Inference, Hypothesis-only Bias, Artificial Patterns 1. Introduction Natural language inference (NLI) (also known as recogniz- ing textual entailment) is a widely studied task which aims to infer the relationship (e.g., entailment, contradiction, neutral) between two fragments of text, known as premise and hypothesis (Dagan et al., 2006; Dagan et al., 2013). NLI models are usually required to determine whether a hy- pothesis is true (entailment) or false (contradiction) given the premise, or whether the truth value can not be in- ferred (neutral). A proper NLI decision should apparently rely on both the premise and the hypothesis. However, some recent studies (Gururangan et al., 2018; Poliak et al., 2018; Tsuchiya, 2018) have shown that it is possible for a trained model to identify the true label by only looking at the hypothesis without observing the premise. The phe- nomenon is referred to as annotation artifacts (Gururangan et al., 2018), statistical irregularities (Poliak et al., 2018) or partial-input heuristics (Feng et al., 2019). In this paper we use the term hypothesis-only bias (Poliak et al., 2018) to refer to this phenomenon. Such hypothesis-only bias originates from the human anno- tation process of data collection. In the data collection pro- cess of many large-scale NLI datasets such as SNLI (Bow- man et al., 2015) and MultiNLI (Williams et al., 2018), hu- man annotators are required to write new sentences (hy- potheses) based on the given premise and a specified label among entailment, contradiction and neutral. Some of the human-elicited hypotheses contain patterns that spuriously correlate to some specific …

Content truncated to 500 words