An Empirical Study on Model-agnostic Debiasing Strategies for Robust Natural Language Inference

Proceedings of the 24th Conference on Computational Natural Language Learning, pages 596–608 Online, November 19-20, 2020. c⃝2020 Association for Computational Linguistics https://doi.org/10.18653/v1/P17 596 An Empirical Study on Model-agnostic Debiasing Strategies for Robust Natural Language Inference Tianyu Liu1 ∗ Xin Zheng3 ∗ Xiaoan Ding4 Baobao Chang1 2 Zhifang Sui1 2 1 Peking University, Beijing, China 2 Peng Cheng Laboratory, Shenzhen, China 3 Beijing University of Posts and Telecommunications, Beijing, China 4 University of Chicago, IL, USA {tianyu0421, chbb, szf}@pku.edu.cn, zheng xin@bupt.edu.cn xiaoanding@uchicago.edu Abstract The prior work on natural language inference (NLI) debiasing mainly targets at one or few known biases while not necessarily making the models more robust. In this paper, we focus on the model-agnostic debiasing strategies and explore how to (or is it possible to) make the NLI models robust to multiple distinct adver- sarial attacks while keeping or even strength- ening the models’ generalization power. We firstly benchmark prevailing neural NLI mod- els including pretrained ones on various adver- sarial datasets. We then try to combat distinct known biases by modifying a mixture of ex- perts (MoE) ensemble method (Clark et al., 2019) and show that it’s nontrivial to miti- gate multiple NLI biases at the same time, and that model-level ensemble method outper- forms MoE ensemble method. We also per- form data augmentation including text swap, word substitution and paraphrase and prove its efficiency in combating various (though not all) adversarial attacks at the same time. Fi- nally, we investigate several methods to merge heterogeneous training data (1.35M) and per- form model ensembling, which are straightfor- ward but effective to strengthen NLI models. 1 Introduction Natural language inference (NLI) (also known as recognizing textual entailment) is a widely stud- ied task which aims to infer the relationship (e.g., entailment, contradiction, neutral) between two fragments of text, known as premise and hypoth- esis (Dagan et al., 2005, 2013). Recent works have found that NLI models are sensitive to the compositional features (Nie et al., 2019), syntactic heuristics (McCoy et al., 2019), stress test (Geiger et al., 2018; Naik et al., 2018) and human ar- tifacts in the data collection phase (Gururangan et al., 2018; Poliak et al., 2018b; Tsuchiya, 2018). ∗Equal contribution. Accordingly, several adversarial datasets are pro- posed for these known biases1. Through our preliminary trials on specific ad- versarial datasets, we find that although the model specific or dataset specific debiasing methods could increase the model performance on the paired adversarial dataset, they might hinder the model performance on other adversarial datasets, as well as hurt the model generalization power, i.e. deficient scores on cross-datasets or cross-domain settings. These phenomena motivate us to investi- gate if it exists a unified model-agnostic debiasing strategy which can mitigate distinct (or even all) known biases while keeping or strengthening the model generalization power. We begin with NLI debiasing models. To make our trials more generic, we adopt a mixture of ex- perts (MoE) strategy (Clark et al., 2019), which is known for being model-agnostic and is adapt- able to …

Content truncated to 500 words