Syntax-BERT: Improving Pre-trained Transformers with Syntax Trees

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 3011–3020 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 3011 Syntax-BERT: Improving Pre-trained Transformers with Syntax Trees Jiangang Bai1,∗, Yujing Wang1,2,†, Yiren Chen1, Yaming Yang2 Jing Bai2, Jing Yu3, Yunhai Tong1 1Peking University, Beijing, China 2Microsoft Research Asia, Beijing, China 3Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China {pku bjg,yujwang,yrchen92,yhtong}@pku.edu.cn {yujwang,yayaming,jbai}@microsoft.com yujing02@iie.ac.cn Abstract Pre-trained language models like BERT achieve superior performances in various NLP tasks without explicit consideration of syn- tactic information. Meanwhile, syntactic in- formation has been proved to be crucial for the success of NLP applications. However, how to incorporate the syntax trees effectively and efficiently into pre-trained Transformers is still unsettled. In this paper, we address this problem by proposing a novel framework named Syntax-BERT. This framework works in a plug-and-play mode and is applicable to an arbitrary pre-trained checkpoint based on Transformer architecture. Experiments on various datasets of natural language un- derstanding verify the effectiveness of syntax trees and achieve consistent improvement over multiple pre-trained models, including BERT, RoBERTa, and T5. 1 Introduction Pre-trained language models like BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2020) and T5 (Raffel et al., 2019) become popular in recent years and achieve outstanding performances in var- ious NLP benchmarks. These models often choose a Transformer architecture largely owing to its at- tractive scalability. Studies (Hewitt and Manning, 2019; Jawahar et al., 2019) have shown that a pre- trained transformer is able to capture certain syn- tactic information implicitly by learning from suf- ficient examples. However, there is still a big gap between the syntactic structures implicitly learned and the golden syntax trees created by human ex- perts. On the other hand, syntax tree is a useful prior for NLP-oriented neural networks (Kiper- wasser and Ballesteros, 2018). For example, ∗The work was done when the author visited Microsoft Research Asia. †Corresponding Author Tree-LSTM (Tai et al., 2015) extends the sequen- tial architecture of LSTM to a tree-structured network. Linguistically-informed self-attention (LISA) (Strubell et al., 2018) proposes a multi- task learning framework for semantic role labeling, which incorporates syntactic knowledge into Trans- former by training one attention head to be attended to its parent in a syntax tree. In addition, Nguyen et al. (2020) integrate tree-structured attention in Transformer with hierarchical accumulation guided by the syntax tree. Although there are numerous works on syntax- enhanced LSTM and Transformer models, none of the previous works have addressed the useful- ness of syntax-trees in the pre-training context. It is straight-forward to ask: it is still helpful to leverage syntax trees explicitly in the pre-training context? If the answer is yes, can we ingest syntax trees into a pre-trained checkpoint efficiently without training from scratch for a specific downstream ap- plication? This is an appealing feature in practice because pre-training from scratch is a huge waste of energy and time. In this paper, we propose Syntax-BERT to tackle the raised questions. Unlike a …

Content truncated to 500 words