Guiding Attention for Self-Supervised Learning with Transformers

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4676–4686 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 4676 Guiding Attention for Self-Supervised Learning with Transformers Ameet Deshpande Karthik Narasimhan Department of Computer Science Princeton University {asd, karthikn}@cs.princeton.edu Abstract In this paper, we propose a simple and ef- fective technique to allow for efficient self- supervised learning with bi-directional Trans- formers. Our approach is motivated by recent studies demonstrating that self-attention pat- terns in trained models contain a majority of non-linguistic regularities. We propose a com- putationally efficient auxiliary loss function to guide attention heads to conform to such pat- terns. Our method is agnostic to the actual pre- training objective and results in faster conver- gence of models as well as better performance on downstream tasks compared to the base- lines, achieving state of the art results in low- resource settings. Surprisingly, we also find that linguistic properties of attention heads are not necessarily correlated with language mod- eling performance.1 1 Introduction Recent advances in self-supervised pre- training (Radford et al., 2018; Devlin et al., 2018a; Liu et al., 2019) have resulted in im- pressive downstream performance on several NLP tasks (Wang et al., 2018, 2019). However, this has led to the development of enormous models, which often require days of training on non-commodity hardware (e.g. TPUs) (Kaplan et al., 2020). Furthermore, studies have shown that it is quite challenging to successfully train these large Transformer models (Vaswani et al., 2017), requiring complicated learning schemes and extensive hyperparameter tuning (Xiong et al., 2020; Raffel et al., 2019; Popel and Bojar, 2018). Despite these expensive training regimes, re- cent studies have found that once trained, these bi-directional language models exhibit simple pat- terns of self-attention without much linguistic back- ing (Voita et al., 2019; Raganato and Tiedemann, 1Code: https://github.com/ameet-1997/AttentionGuidance Attention Guided Model RoBERTa-Only MLM 0% 1% 100% Figure 1: Attention patterns of our model (left) and the default RoBERTa model (right) after 0% (top), 1% (middle) and 100% (bottom) of pre-training. Induc- ing simple patterns (left) using an auxiliary loss leads to benefits in convergence speed, downstream perfor- mance, and robustness to hyperparameters. 2018). For example, 40% of heads in a pre-trained BERT model (Devlin et al., 2018a) simply pay at- tention to delimiters added by the tokenizer (e.g. [CLS] or [SEP]) (Kovaleva et al., 2019). Since these attention patterns are independent of linguis- tic phenomena, a natural question arises: can Trans- former models be guided towards such attention patterns without requiring extensive training? In this paper, we propose an attention guid- ance (AG) mechanism for self-attention modules in Transformer architectures to enable faster, more efficient, and robust self-supervised learning. Our approach is simple and agnostic to the training ob- jective. Specifically, we introduce an auxiliary loss function to guide the self-attention heads in each layer towards a set of pre-determined patterns (e.g. Figure 1 (Vig, 2019)). These patterns encourage the formation of both global (e.g. attend to [CLS], [SEP] tokens) and local (e.g. attend to [Next], …

Content truncated to 500 words