Towards Interpreting and Mitigating Shortcut Learning Behavior of NLU models

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 915–929 June 6–11, 2021. ©2021 Association for Computational Linguistics 915 Towards Interpreting and Mitigating Shortcut Learning Behavior of NLU Models Mengnan Du1∗, Varun Manjunatha2, Rajiv Jain2, Ruchi Deshpande3, Franck Dernoncourt2, Jiuxiang Gu2, Tong Sun2 and Xia Hu1 1Texas A&M University 2Adobe Research 3Adobe Document Cloud dumengnan@tamu.edu {vmanjuna,rajijain,rdeshpan,franck.dernoncourt,jigu,tsun}@adobe.com, hu@cse.tamu.edu Abstract Recent studies indicate that NLU models are prone to rely on shortcut features for predic- tion, without achieving true language under- standing. As a result, these models fail to gen- eralize to real-world out-of-distribution data. In this work, we show that the words in the NLU training set can be modeled as a long- tailed distribution. There are two findings: 1) NLU models have strong preference for fea- tures located at the head of the long-tailed dis- tribution, and 2) Shortcut features are picked up during very early few iterations of the model training. These two observations are further employed to formulate a measurement which can quantify the shortcut degree of each training sample. Based on this shortcut mea- surement, we propose a shortcut mitigation framework LTGR, to suppress the model from making overconfident predictions for samples with large shortcut degree. Experimental re- sults on three NLU benchmarks demonstrate that our long-tailed distribution explanation ac- curately reflects the shortcut learning behavior of NLU models. Experimental analysis further indicates that LTGR can improve the general- ization accuracy on OOD data, while preserv- ing the accuracy on in-distribution data. 1 Introduction Pre-trained language models, such as BERT (De- vlin et al., 2019), have demonstrated substantial gains on many NLU (natural language understand- ing) benchmarks. However, recent studies show that these models tend to exploit dataset biases as shortcuts to make predictions, rather than learn the semantic understanding and reasoning (Geirhos et al., 2020; Gururangan et al., 2018). Here we fo- cus on the lexical bias, where NLU models rely on spurious correlations between shortcut words and ∗Most of the work was done while the first author was an intern at Adobe Research. labels. This eventually results in their low general- izability on out-of-distribution (OOD) samples and low adversarial robustness (Zellers et al., 2018). In this work, we show that the shortcut learning behavior of NLU models can be explained by the long-tailed phenomenon. Previous empirical anal- ysis indicates that the performance of BERT-like models for NLI task could be mainly explained by the reliance of spurious statistical cues such as unigrams “not”, “do”, “is” and bigrams “will not” (Niven and Kao, 2019; Gururangan et al., 2018). Here we generalize these hypotheses us- ing the long-tailed phenomenon. Specifically, the features in training set could be modeled using a long-tailed distribution via using local mutual in- formation (Evert, 2005) as a measurement. By utilizing an interpretation method to analyze model behavior, we observe that these NLU models con- centrate mainly on information on the head of the distribution, which usually corresponds to non- generalizable shortcut …

Content truncated to 500 words