Convolutional Self-Attention Networks

Proceedings of NAACL-HLT 2019, pages 4040–4045 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 4040 Convolutional Self-Attention Networks Baosong Yang† Longyue Wang‡ Derek F. Wong† Lidia S. Chao† Zhaopeng Tu‡∗ †NLP2CT Lab, Department of Computer and Information Science, University of Macau ‡Tencent AI Lab †nlp2ct.baosong@gmail.com, {derekfw,lidiasc}@umac.mo ‡{vinnylywang,zptu}@tencent.com Abstract Self-attention networks (SANs) have drawn increasing interest due to their high paral- lelization in computation and flexibility in modeling dependencies. SANs can be fur- ther enhanced with multi-head attention by allowing the model to attend to information from different representation subspaces. In this work, we propose novel convolutional self-attention networks, which offer SANs the abilities to 1) strengthen dependencies among neighboring elements, and 2) model the inter- action between features extracted by multiple attention heads. Experimental results of ma- chine translation on different language pairs and model settings show that our approach outperforms both the strong Transformer base- line and other existing models on enhancing the locality of SANs. Comparing with prior studies, the proposed model is parameter free in terms of introducing no more parameters. 1 Introduction Self-attention networks (SANs) (Parikh et al., 2016; Lin et al., 2017) have shown promising empirical results in various natural language pro- cessing (NLP) tasks, such as machine transla- tion (Vaswani et al., 2017), natural language in- ference (Shen et al., 2018a), and acoustic model- ing (Sperber et al., 2018). One appealing strength of SANs lies in their ability to capture dependen- cies regardless of distance by explicitly attending to all the elements. In addition, the performance of SANs can be improved by multi-head atten- tion (Vaswani et al., 2017), which projects the in- put sequence into multiple subspaces and applies attention to the representation in each subspace. Despite their success, SANs have two major limitations. First, the model fully take into ac- ∗Zhaopeng Tu is the corresponding author of the paper. This work was conducted when Baosong Yang was interning at Tencent AI Lab. count all the elements, which disperses the atten- tion distribution and thus overlooks the relation of neighboring elements and phrasal patterns (Yang et al., 2018; Wu et al., 2018; Guo et al., 2019). Second, multi-head attention extracts distinct lin- guistic properties from each subspace in a parallel fashion (Raganato and Tiedemann, 2018), which fails to exploit useful interactions across differ- ent heads. Recent work shows that better features can be learned if different sets of representations are present at feature learning time (Ngiam et al., 2011; Lin et al., 2014). To this end, we propose novel convolutional self-attention networks (CSANs), which model lo- cality for self-attention model and interactions be- tween features learned by different attention heads in an unified framework. Specifically, in order to pay more attention to a local part of the input se- quence, we restrict the attention scope to a window of neighboring elements. The localness is there- fore enhanced via a parameter-free 1-dimensional convolution. Moreover, we extend the convolu- tion to a 2-dimensional area with the axis of at- tention …

Content truncated to 500 words