Towards Domain-Generalizable Paraphrase Identification by Avoiding the Shortcut Learning

Proceedings of Recent Advances in Natural Language Processing, pages 1318–1325 Sep 1–3, 2021. https://doi.org/10.26615/978-954-452-072-4_148 1318 Towards Domain-Generalizable Paraphrase Identification by Avoiding the Shortcut Learning Xin Shen The Chinese University of Hong Kong xshen@se.cuhk.edu.hk Wai Lam The Chinese University of Hong Kong wlam@se.cuhk.edu.hk Abstract In this paper, we investigate the Domain Ge- neralization (DG) problem for supervised Pa- raphrase Identification (PI). We observe that the performance of existing PI models dete- riorates dramatically when tested in an out- of-distribution (OOD) domain. We conjecture that it is caused by shortcut learning, i.e., the- se models tend to utilize the cue words that are unique for a particular dataset or domain. To alleviate this issue and enhance the DG ability, we propose a PI framework based on Optimal Transport (OT). Our method forces the network to learn the necessary features for all the words in the input, which allevia- tes the shortcut learning problem. Experimen- tal results show that our method improves the DG ability for the PI models. 1 Introduction Paraphrase Identification (PI) is the task of reco- gnizing whether one text is a restatement of ano- ther text, preserving the same meaning while ad- opting a different expression (Bhagat and Hovy, 2013). Neural network based models have been proposed for the supervised PI task, and achie- ve decent performance in the single-domain set- ting (Yin and Schütze, 2015; Wang et al., 2017; Yang et al., 2019). At present, the existing PI corpora are restricted to several particular do- mains (Dolan et al., 2004; Xu et al., 2014; He et al., 2020), while the practical sentence pair for the pa- raphrase judgment can be from any unlabeled do- main. At the same time, building a PI corpus for a novel domain needs massive human effort and is expensive. Therefore, a natural question is: for the supervised models trained in the domains with annotated PI corpora, to what extent can they ge- neralize to an out-of-distribution (OOD) domain? In this paper, we investigate the multi- source (Blanchard et al., 2011) Domain Generali- zation (DG) (Wang et al., 2021; Zhou et al., 2021) problem for supervised PI. More specifically, we try to learn a PI model based on information from several annotated source domains, and it could ge- neralize well to an unlabeled domain. We inves- tigate several competitive PI models in the DG setting, and observe that their performance dete- riorates dramatically when tested in an OOD do- main. We conjecture that the poor performance is caused by the models’ tendency to the shortcut learning (Geirhos et al., 2020). More specifically, these models are prone to relying on the shortcut features, e.g., some cue words, for classification. These shortcut features are often unique in one par- ticular dataset or domain. When tested on an OOD domain, the models’ performance deteriorates be- cause the shortcuts are missing. Interestingly, this phenomenon is also observed in other NLP tasks or models, such as NLI (Gururangan et al., 2018; Du et al., 2021), reading comprehension (Kaushik and …

Content truncated to 500 words