Enhancing Generalization in Natural Language Inference by Syntax

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4973–4978 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 4973 Enhancing Generalization in Natural Language Inference by Syntax Qi He1∗, Han Wang2∗, Yue Zhang3,4† 1 University of Electronic Science and Technology of China, Chengdu, China 2 New York University, New York, USA 3 School of Engineering, Westlake University, Hangzhou, China 4 Institute of Advanced Technology, Westlake Institute for Advanced Study, Hangzhou, China heqi@uestc.edu.cn, hwang@nyu.edu, zhangyue@westlake.edu.cn Abstract Pre-trained language models such as BERT have achieved the state-of-the-art performance on natural language inference (NLI). How- ever, it has been shown that such models can be tricked by variations of surface patterns such as syntax. We investigate the use of dependency trees to enhance the generaliza- tion of BERT in the NLI task, leveraging on a graph convolutional network to represent a syntax-based matching graph with heteroge- neous matching patterns. Experimental results show that, our syntax-based method largely enhance generalization of BERT on a test set where the sentence pair has high lexical over- lap but diverse syntactic structures, and do not degrade performance on the standard test set. In other words, the proposed method makes BERT more robust on syntactic changes. 1 Introduction The task of natural language inference (NLI) tar- gets at determining whether one sentence entails another (Condoravdi et al., 2003). Recently, large- scale pre-trained contextualized embeddings such as BERT (Devlin et al., 2018) and XLNet (Yang et al., 2019) have given the state-of-the-art accuracy for this task. It has been shown that pre-trained models help to better capture heuristic patterns in a set of training data and therefore enhance in- domain performance (Wang et al., 2018). However, there are still limitations on the generalization of such models to examples under a different distribu- tion. In particular, it has been shown that seemingly simple types of examples in a carefully designed evaluation set (i.e. HANS) can lead to significant degeneration and large variability in performance ∗Equal Contribution. Work is done when working at West- lake University. †Corresponding author. (McCoy et al., 2019b,a). Table 1 shows a set of test cases from HANS, where premise and hypothesis have high lexical overlap but different syntactic structures. The BERT model gives incorrect results on most cases. This issue can negatively affect NLI applications such as dialogue (Dziri et al., 2019; Welleck et al., 2019). It has been shown that syntactic structures are useful for cross-domain generalization of NLP models (Wang et al., 2017; Strubell and McCallum, 2018). Intuitively, a more robust NLI model can be obtained by making use of structural information. We empirically investigate the effectiveness of syn- tactic features for enhancing the generalization of BERT-based matching models. In particular, given a pair of sentences, the dependency syntax of each sentence is obtained using a neural parser (Qi et al., 2018). The parse trees are then extended using four types of edge patterns, including a soft co-attention matching pattern that links the sentence pair into an integrated graph. A graph …

Content truncated to 500 words