Improving BERT with Syntax-aware Local Attention

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 645–653 August 1–6, 2021. ©2021 Association for Computational Linguistics 645 Improving BERT with Syntax-aware Local Attention Zhongli Li1∗, Qingyu Zhou2†, Chao Li2, Ke Xu1, Yunbo Cao2 1Beihang University 2Tencent Cloud Xiaowei {lizhongli@,kexu@nlsde.}buaa.edu.cn {qingyuzhou,diegoli,yunbocao}@tencent.com Abstract Pre-trained Transformer-based neural lan- guage models, such as BERT, have achieved remarkable results on varieties of NLP tasks. Recent works have shown that attention-based models can benefit from more focused atten- tion over local regions. Most of them restrict the attention scope within a linear span, or confine to certain tasks such as machine trans- lation and question answering. In this pa- per, we propose a syntax-aware local atten- tion, where the attention scopes are restrained based on the distances in the syntactic struc- ture. The proposed syntax-aware local atten- tion can be integrated with pretrained language models, such as BERT, to render the model to focus on syntactically relevant words. We con- duct experiments1 on various single-sentence benchmarks, including sentence classification and sequence labeling tasks. Experimental re- sults show consistent gains over BERT on all benchmark datasets. The extensive studies ver- ify that our model achieves better performance owing to more focused attention over syntacti- cally relevant words. 1 Introduction Recently, Transformer (Vaswani et al., 2017) has performed remarkably well, standing on the multi- headed dot-product attention which fully takes into account the global contextualized informa- tion. Several studies find that self-attention can be enhanced by local attention, where the atten- tion scopes are restricted to important local regions. Luong et al. (2015); Yang et al. (2018); Xu et al. (2019); Nguyen et al. (2020) utilize dynamic or fixed windows to perform local attention. Strubell ∗Contribution done during internship at Tencent Cloud Xiaowei. †Corresponding author. 1The code is available at https://github.com/ Neutralzz/syntax_aware_local_attention et al. (2018); Zhang et al. (2020); Bugliarello and Okazaki (2020) explore to utilize syntax to restrain attention for better performance, but each of them confines to a certain task. In this work, we propose a syntax-aware local attention (SLA) which is adaptable to several tasks, and integrate it with BERT (Devlin et al., 2019). We first apply dependency parsing to the input text, and calculate the distances of input words to con- struct the self-attention masks. The local attention scores are calculated by applying these masks to the dot-product attention. Then we incorporate the syntax-aware local attention with the Transformer global attention. A gate unit is employed for each token in each layer, which determines how much attention is paid to syntactically relevant words. We lift weights from existing pre-trained BERT, and evaluate our models on several single-sentence benchmarks, including sentence classification and sequence labeling tasks. Experimental results show that our method achieves consistent performance gains over BERT and outperforms previous syntax- based approaches on the average performance. Fur- thermore, we compare our syntax-aware local at- tention with the window-based local attention. We find that the syntax-aware local attention is more involved in the aggregation of local and global at- tention. The …

Content truncated to 500 words