Fine-tune BERT with Sparse Self-Attention Mechanism

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3548–3553, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 3548 Fine-tune BERT with Sparse Self-Attention Mechanism Baiyun Cui, Yingming Li∗, Ming Chen, and Zhongfei Zhang College of Information Science and Electronic Engineering, Zhejiang University, China baiyunc@yahoo.com,{yingming,funkyblack,zhongfei}@zju.edu.cn Abstract In this paper, we develop a novel Sparse Self-Attention Fine-tuning model (referred as SSAF) which integrates sparsity into self- attention mechanism to enhance the fine- tuning performance of BERT. In particular, sparsity is introduced into the self-attention by replacing softmax function with a control- lable sparse transformation when fine-tuning with BERT. It enables us to learn a structurally sparse attention distribution, which leads to a more interpretable representation for the whole input. The proposed model is evalu- ated on sentiment analysis, question answer- ing, and natural language inference tasks. The extensive experimental results across multiple datasets demonstrate its effectiveness and su- periority to the baseline methods. 1 Introduction Recently, the pre-trained language models obtain new state-of-the-art results on a broad range of tasks, for example ULMFiT (Howard and Ruder, 2018), ELMo (Peters et al., 2018), and Ope- nAI GPT (Radford et al., 2018). Devlin et al. (2018) proposed BERT, a deep bidirectional lan- guage representation model which substantially outperforms the previous methods and has at- tracted wide attention in natural language process- ing. Fine-tuning on the pre-trained BERT has shown to be beneficial to improve many down- stream tasks such as document classification (Ad- hikari et al., 2019), extractive summarization (Liu, 2019), question answering (Yang et al., 2019), nat- ural language inference (Liu et al., 2019), and reading comprehension (Xu et al., 2019). To further understand the impact of BERT on the fine-tuning performance, we exploit the atten- tion behavior of the model when fine-tuning BERT ∗Corresponding author Figure 1: The explorations of attention behavior for four models: No pre-training, Fine-tuning BERT, SSAF(No pre-training), and SSAF. All the attention matrices are extracted from the eighth attention layer of the model for the sentence in Sentube-A dataset. Words on the y-axis are attending to the words on the x-axis. The attention distribution in SSAF presents more struc- turally sparse patterns compared to the others. in sentiment analysis task. In particular, the at- tention matrices of models with different setups are visualized in Figure 1. As shown in Figure 1(b), the attention distribution of fine-tuning with BERT is close to being a band matrix, with interest concentrated on small regions near the diagonal. Compared to the model trained from scratch with- out pre-training (Figure 1(a)), more interpretabil- ity would be provided by fine-tuning with BERT due to its structural attention distribution. This dif- ference in the distributions might be caused by the masked language model task of BERT, where the model is required to predict the original vocabu- 3549 lary id of the masked token based on its context. Consequently, the local interactions among differ- ent words are strengthened …

Content truncated to 500 words