Convolutions and Self-Attention: Re-interpreting Relative Positions in Pre-trained Language Models

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 4322–4333 August 1–6, 2021. ©2021 Association for Computational Linguistics 4322 Convolutions and Self-Attention: Re-interpreting Relative Positions in Pre-trained Language Models Tyler A. Chang1,2, Yifan Xu1, Weijian Xu1, Zhuowen Tu1 1University of California San Diego 2Halıcıo˘glu Data Science Institute {tachang, yix081, wex041, ztu}@ucsd.edu Abstract In this paper, we detail the relationship be- tween convolutions and self-attention in nat- ural language tasks. We show that relative position embeddings in self-attention layers are equivalent to recently-proposed dynamic lightweight convolutions, and we consider multiple new ways of integrating convolutions into Transformer self-attention. Specifically, we propose composite attention, which unites previous relative position embedding meth- ods under a convolutional framework. We conduct experiments by training BERT with composite attention, finding that convolutions consistently improve performance on multi- ple downstream tasks, replacing absolute posi- tion embeddings. To inform future work, we present results comparing lightweight convo- lutions, dynamic convolutions, and depthwise- separable convolutions in language model pre- training, considering multiple injection points for convolutions in self-attention layers. 1 Introduction In recent years, Transformer-based language mod- els have brought dramatic improvements on a wide range of natural language tasks (Brown et al., 2020; Devlin et al., 2019). The central innovation of Transformer architectures is the self-attention mechanism (Vaswani et al., 2017), which has grown beyond NLP, extending into domains rang- ing from computer vision (Dosovitskiy et al., 2021) and speech recognition (Dong et al., 2018) to rein- forcement learning (Parisotto et al., 2020; Touvron et al., 2020). In computer vision, self-attention and convolu- tions have been combined to achieve competitive results for image classification (Bello et al., 2019). Similarly, researchers in NLP have begun integrat- ing convolutions into self-attention for natural lan- guage tasks. Recent work has shown initial success adding convolutional modules to self-attention in pre-trained language models (Jiang et al., 2020), or even replacing self-attention entirely with dynamic convolutions (Wu et al., 2019). These successes defy theoretical proofs showing that multi-headed self-attention with relative position embeddings is strictly more expressive than convolution (Cordon- nier et al., 2020). To identify why convolutions have been successful in NLP, we seek to isolate the differences between self-attention and convolution in the context of natural language. In this work, we formalize the relationship between self-attention and convolution in Trans- former encoders by generalizing relative position embeddings, and we identify the benefits of each approach for language model pre-training. We show that self-attention is a type of dynamic lightweight convolution, a data-dependent convo- lution that ties weights across input channels (Wu et al., 2019). Notably, previous methods of en- coding relative positions (Shaw et al., 2018; Raf- fel et al., 2020) are direct implementations of lightweight convolutions. Under our framework, the benefits of convolution come from an ability to capture local position information in sentences. Then, we propose composite attention, which ap- plies a lightweight convolution that combines previ- ous relative position embedding methods. We find that …

Content truncated to 500 words