Syntax-Enhanced Pre-trained Model

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 5412–5422 August 1–6, 2021. ©2021 Association for Computational Linguistics 5412 Syntax-Enhanced Pre-trained Model Zenan Xu1∗ †, Daya Guo1∗, Duyu Tang2†, Qinliang Su1,4,5‡, Linjun Shou3, Ming Gong3, Wanjun Zhong1∗, Xiaojun Quan1, Daxin Jiang3, and Nan Duan2 1School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, China 2Microsoft Research Asia, Beijing, China 3Microsoft Search Technology Center Asia, Beijing, China 4Guangdong Key Laboratory of Big Data Analysis and Processing, Guangzhou, China 5Key Lab. of Machine Intelligence and Advanced Computing, Ministry of Education, China {xuzn, guody5, zhongwj25}@mail2.sysu.edu.cn {suqliang, quanxj3}@mail.sysu.edu.cn {dutang,lisho,migon,djiang,nanduan}@microsoft.com Abstract We study the problem of leveraging the syn- tactic structure of text to enhance pre-trained models such as BERT and RoBERTa. Exist- ing methods utilize syntax of text either in the pre-training stage or in the fine-tuning stage, so that they suffer from discrepancy between the two stages. Such a problem would lead to the necessity of having human-annotated syntactic information, which limits the appli- cation of existing methods to broader scenar- ios. To address this, we present a model that utilizes the syntax of text in both pre-training and fine-tuning stages. Our model is based on Transformer with a syntax-aware attention layer that considers the dependency tree of the text. We further introduce a new pre-training task of predicting the syntactic distance among tokens in the dependency tree. We evaluate the model on three downstream tasks, including relation classification, entity typing, and ques- tion answering. Results show that our model achieves state-of-the-art performance on six public benchmark datasets. We have two ma- jor findings. First, we demonstrate that in- fusing automatically produced syntax of text improves pre-trained models. Second, global syntactic distances among tokens bring larger performance gains compared to local head re- lations between contiguous tokens.1 1 Introduction Pre-trained models such as BERT (Devlin et al., 2019), GPT (Radford et al., 2018), and RoBERTa (Liu et al., 2019) have advanced the state-of-the-art performances of various natural language process- ing tasks. The successful recipe is that a model is first pre-trained on a huge volume of unsupervised ∗ Work is done during internship at Microsoft. † For questions, please contact D. Tang and Z. Xu. ‡ Corresponding author. 1The source data is available at https://github.com/Hi- ZenanXu/Syntax-Enhanced Pre-trained Model. data with self-supervised objectives, and then is fine-tuned on supervised data with the same data scheme. Dominant pre-trained models represent a text as a sequence of tokens2. The merits are that such basic text representations are available from vast amounts of unsupervised data, and that models pre-trained and fine-tuned with the same paradigm usually achieve good accuracy in practice (Guu et al., 2020). However, an evident limitation of these methods is that richer syntactic structure of text is ignored. In this paper, we seek to enhance pre-trained models with syntax of text. Related studies attempt to inject syntax information either only in the fine- tuning stage (Nguyen et al., 2020; …

Content truncated to 500 words