Towards Robustifying NLI Models Against Lexical Dataset Biases

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8759–8771 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 8759 Towards Robustifying NLI Models Against Lexical Dataset Biases Xiang Zhou Mohit Bansal UNC Chapel Hill {xzh, mbansal}@cs.unc.edu Abstract While deep learning models are making fast progress on the task of Natural Language In- ference, recent studies have also shown that these models achieve high accuracy by exploit- ing several dataset biases, and without deep un- derstanding of the language semantics. Using contradiction-word bias and word-overlapping bias as our two bias examples, this paper ex- plores both data-level and model-level debias- ing methods to robustify models against lexi- cal dataset biases. First, we debias the dataset through data augmentation and enhancement, but show that the model bias cannot be fully removed via this method. Next, we also com- pare two ways of directly debiasing the model without knowing what the dataset biases are in advance. The first approach aims to remove the label bias at the embedding level. The second approach employs a bag-of-words sub- model to capture the features that are likely to exploit the bias and prevents the original model from learning these biased features by forcing orthogonality between these two sub- models. We performed evaluations on new balanced datasets extracted from the original MNLI dataset as well as the NLI stress tests, and show that the orthogonality approach is better at debiasing the model while maintain- ing competitive overall accuracy.1 1 Introduction In this work, we focus on investigating and re- ducing biases in the task of Natural Language In- ference (NLI), where the target of the model is to classify the relations between a pair of sen- tences into three categories: entailment, neutral and contradiction. With the release of large-scale standard datasets (Bowman et al., 2015; Williams et al., 2018), significant success has been made on 1Our code and data are available at: https://github. com/owenzx/LexicalDebias-ACL2020 this task, and recent state-of-the-art neural mod- els have already reached competitive performance even compared to humans. However, a number of papers (Gururangan et al., 2018; Poliak et al., 2018; Nie et al., 2019; Naik et al., 2018) have shown that despite the high accuracy on these datasets, these models are far from mastering the required nature of natural language inference. Instead of deeply understanding the sentences in the correct semantic way, these models tend to exploit shortcuts or an- notation artifacts in the dataset and actually overfit to these datasets to predict the label using sim- ple patterns. However, most shortcuts are only valid within the datasets and fail to hold for gen- eral natural language. Hence, these models fail to generalize to other datasets for the same task (Tal- man and Chatzikyriakidis, 2019), perform badly on challenge analysis datasets (Glockner et al., 2018; McCoy et al., 2019; Wang et al., 2019b), and are fooled by adversarial attacks (Naik et al., 2018). One major cause of this problem is the existence of dataset biases. Since most …

Content truncated to 500 words