Towards the Necessity for Debiasing Natural Language Inference Datasets

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 6883–6888 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 6883 Towards the Necessity for Debiasing Natural Language Inference Datasets Mithun Paul Panenghat, Sandeep Suntwal, Faiz Rafique, Rebecca Sharp, Mihai Surdeanu Department of Computer Science, University of Arizona Tucson, Arizona {mithunpaul, sandeepsuntwal, faizr, bsharp, msurdeanu}@email.arizona.edu Abstract Modeling natural language inference is a challenging task. With large annotated data sets available it has now become feasible to train complex neural network based inference methods which achieve state of the art performance. However, it has been shown that these models also learn from the subtle biases inherent in these datasets (Gururangan et al., 2018). In this work we explore two techniques for delexicalization that modify the datasets in such a way that we can control the importance that neural-network based methods place on lexical entities. We demonstrate that the proposed methods not only maintain the performance in-domain but also improve performance in some out-of-domain settings. For example, when using the delexicalized version of the FEVER dataset, the in-domain performance of a state of the art neural network method dropped only by 1.12% while its out-of-domain performance on the FNC dataset improved by 4.63%. We release the delexicalized versions of three common datasets used in natural language inference. These datasets are delexicalized using two methods: one which replaces the lexical entities in an overlap-aware manner, and a second, which additionally incorporates semantic lifting of nouns and verbs to their WordNet hypernym synsets. 1. Introduction The task of natural language inference (NLI) is considered to be an integral part of natural language understanding (NLU). In this task, which can be seen as a particular in- stance of recognizing textual entailment (RTE) (Fyodorov et al., 2000; Condoravdi et al., 2003; Bos and Markert, 2005; MacCartney and Manning, 2009), a model is asked to classify if a given sentence (premise) entails, contradicts or is neutral given a second sentence (hypothesis). In order to advance any task in natural language process- ing (NLP), quality data sets are quintessential. Some such data sets which have enabled the advancement of NLI (and fact verification) are SNLI (Bowman et al., 2015) MNLI (Williams et al., 2017), FEVER (Thorne et al., 2018), and FNC (Pomerleau and Rao, 2017). However these datasets are not devoid of biases (subtle sta- tistical patterns in a dataset, which could have been intro- duced either due to the methodology of data collection or due to an inherent social bias). For example, (Gururan- gan et al., 2018) and Poliak et al. (2018) show that bi- ases were introduced into the MNLI dataset by certain lan- guage creation choices made by the crowd workers. Sim- ilarly, Schuster et al. (2019) show that in the FEVER, the REFUTES label (same as the contradicts label men- tioned above) highly correlates with the presence of nega- tion phrases. These biases can be readily exploited by neural networks (NNs), and thus have influence on performance. …

Content truncated to 500 words