Syntax-guided Contrastive Learning for Pre-trained Language Model

Findings of the Association for Computational Linguistics: ACL 2022, pages 2430 - 2440 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Syntax-guided Contrastive Learning for Pre-trained Language Model Shuai Zhang, Lijie Wang, Xinyan Xiao, Hua Wu Baidu Inc, Beijing, China {zhangshuai28, wanglijie, xiaoxinyan, wu_hua}@baidu.com Abstract Syntactic information has been proved to be useful for transformer-based pre-trained lan- guage models. Previous studies often rely on additional syntax-guided attention components to enhance the transformer, which require more parameters and additional syntactic parsing in downstream tasks. This increase in complex- ity severely limits the application of syntax- enhanced language model in a wide range of scenarios. In order to inject syntactic knowl- edge effectively and efficiently into pre-trained language models, we propose a novel syntax- guided contrastive learning method which does not change the transformer architecture. Based on constituency and dependency structures of syntax trees, we design phrase-guided and tree- guided contrastive objectives, and optimize them in the pre-training stage, so as to help the pre-trained language model to capture rich syn- tactic knowledge in its representations. Experi- mental results show that our contrastive method achieves consistent improvements in a variety of tasks, including grammatical error detection, entity tasks, structural probing and GLUE. De- tailed analysis further verifies that the improve- ments come from the utilization of syntactic information, and the learned attention weights are more explainable in terms of linguistics. 1 Introduction Pre-trained transformer-based neural language models (LMs), such as BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019), have achieved re- markable results in a variety of NLP tasks (Wang et al., 2018). However, many studies have found that these LMs do not encode enough syntactic knowledge in their learned representations (Wang et al., 2019; Min et al., 2020; Wang et al., 2020). As it is widely acknowledged that structural informa- tion is very important for NLP (Strubell et al., 2018; Nguyen et al., 2019; Zhang et al., 2020), there is an increasing interest in improving pre-trained LMs by using syntactic information. Most of these works enhance pre-trained LMs by adding syntax-driven attention components to the transformer (Li et al., 2020b; Xu et al., 2020; Bai et al., 2021). They use the added components to produce a syntax-aware representation, and in- ject this additional representation into the original one from the vanilla transformer, so as to get a final syntax-enhanced representation. Although these works did bring improvements, the additional syntax-aware layers obviously increase application inconvenience and computation complexity, as they need to parse the input text during testing and re- quire more neural parameters. Moreover, the per- formance of such explicit method depends on the parsing quality of test data (Sachan et al., 2020). There are also some efforts on incorporating syntax- related objectives into the pre-training stage, such as syntax head prediction (Wang et al., 2020) and dependency distance prediction (Xu et al., 2020). However, these predictive pre-training tasks often fail to improve performance alone and need to work together with the additional attention components (Xu et al., …

Content truncated to 500 words