Interventional Training for Out-Of-Distribution Natural Language Understanding

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11627 - 11638 December 7-11, 2022 ©2022 Association for Computational Linguistics Interventional Training for Out-Of-Distribution Natural Language Understanding Sicheng Yu1,2∗ Jing Jiang1 Hao Zhang3 Yulei Niu4 Qianru Sun1 Lidong Bing† 2 1Singapore Management University 2DAMO Academy, Alibaba Group 3Nanyang Technological University 4Columbia University scyu.2018@phdcs.smu.edu.sg, {jingjiang,qianrusun}@smu.edu.sg hzhang26@outlook.com, yn.yuleiniu@gmail.com l.bing@alibaba-inc.com Abstract Out-of-distribution (OOD) settings are used to measure a model’s performance when the dis- tribution of the test data is different from that of the training data. NLU models are known to suffer in OOD settings (Utama et al., 2020b). We study this issue from the perspective of causality, which sees confounding bias as the reason for models to learn spurious correlations. While a common solution is to perform inter- vention, existing methods handle only known and single confounder (Pearl and Mackenzie, 2018), but in many NLU tasks the confounders can be both unknown and multifactorial. In this paper, we propose a novel interventional training method called Bottom-up Automatic Intervention (BAI) that performs multi-granular intervention with identified multifactorial con- founders. Our experiments on three NLU tasks, namely, natural language inference, fact verifi- cation and paraphrase identification, show the effectiveness of BAI for tackling different OOD settings. 1 1 Introduction From the era of word embeddings (Pennington et al., 2014) to pre-trained language models (Devlin et al., 2019), researchers of natural language under- standing (NLU) have tried to push the performance on benchmark datasets. Traditional settings as- sume independent and identical distribution (IID) in training and testing splits. However, the IID set- ting cloaks the vulnerability of neural models, i.e., neural models tend to learn non-robust “shortcut” patterns in the training data but fail to make robust predictions on unseen samples. To evaluate the ro- bustness of models, the out-of-distribution (OOD) setting draws the attention of the NLU community. ∗ The work was done when Sicheng was an intern at Alibaba and Yulei was at Nanyang Technological University. † Corresponding author. 1Our code is publicly available on GitHub: https:// github.com/PluviophileYU/BAI 0% 20% 40% 60% 80% 100% IID#Data#(MNLI) 0% 20% 40% 60% 80% 100% 0~20% 21~40% 41~60% 61~80% 81~100% OOD#Data#(HANS) Lexical#Overlap Figure 1: The proportions of entailment and non- entailment samples with different percentages of lexical overlap. For example, the task of natural language infer- ence (NLI) determines whether a hypothesis can be entailed from a premise. We can observe that the lexical overlap between the hypothesis and the premise correlates with the entailment label on the benchmark MNLI dataset (Williams et al., 2018) (as shown in the top part of Figure 1). McCoy et al. (2019) proposed an OOD set named HANS for NLI. As shown in the bottom part of Figure 1, HANS does not have the correlation between lexi- cal overlap and the entailment label. NLI models that rely on the lexical overlap heuristic suffer from a significant degradation on HANS (Utama et al., 2020b). Recently, causal inference has been adopted in NLP to identify robust correlations by …

Content truncated to 500 words