Information Aggregation for Multi-Head Attention with Routing-by-Agreement

Proceedings of NAACL-HLT 2019, pages 3566–3575 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 3566 Information Aggregation for Multi-Head Attention with Routing-by-Agreement Jian Li1 Baosong Yang2 Zi-Yi Dou3 Xing Wang4 Michael R. Lyu1 Zhaopeng Tu4∗ 1Department of Computer Science and Engineering, The Chinese University of Hong Kong {jianli,lyu}@cse.cuhk.edu.hk 2University of Macau 3Carnegie Mellon University nlp2ct.baosong@gmail.com zdou@andrew.cmu.edu 4Tencent AI Lab {brightxwang,zptu}@tencent.com Abstract Multi-head attention is appealing for its abil- ity to jointly extract different types of in- formation from multiple representation sub- spaces. Concerning the information aggre- gation, a common practice is to use a con- catenation followed by a linear transformation, which may not fully exploit the expressiveness of multi-head attention. In this work, we pro- pose to improve the information aggregation for multi-head attention with a more powerful routing-by-agreement algorithm. Specifically, the routing algorithm iteratively updates the proportion of how much a part (i.e. the distinct information learned from a specific subspace) should be assigned to a whole (i.e. the fi- nal output representation), based on the agree- ment between parts and wholes. Experimen- tal results on linguistic probing tasks and ma- chine translation tasks prove the superiority of the advanced information aggregation over the standard linear transformation. 1 Introduction Attention model becomes a standard component of the deep learning networks, contributing to impressive results in machine translation (Bah- danau et al., 2015; Luong et al., 2015), image captioning (Xu et al., 2015), speech recogni- tion (Chorowski et al., 2015), among many other applications. Its superiority lies in the ability of modeling the dependencies between representa- tions without regard to their distance. Recently, the performance of attention is further improved by multi-head mechanism (Vaswani et al., 2017), which parallelly performs attention functions on different representation subspaces of the input se- quence. Consequently, different attention heads are able to capture distinct linguistic properties of ∗ Zhaopeng Tu is the corresponding author of the pa- per. This work was mainly conducted when Jian Li, Baosong Yang, and Zi-Yi Dou were interning at Tencent AI Lab. the input, which are embedded in different sub- spaces (Raganato and Tiedemann, 2018). Subse- quently, a linear transformation is generally em- ployed to aggregate the partial representations ex- tracted by different attention heads (Vaswani et al., 2017; Ahmed et al., 2018). Most existing work focus on extracting in- formative or distinct partial-representations from different subspaces (e.g. Lin et al., 2017; Li et al., 2018), while few studies have paid atten- tion to the aggregation of the extracted partial- representations. Arguably, information extraction and aggregation are both crucial for multi-head at- tention to generate an informative representation. Recent studies in multimodal learning show that a straightforward linear transformation for fusing features in different sets of representations usu- ally limits the extent of abstraction (Fukui et al., 2016; Ben-Younes et al., 2017). A natural ques- tion arises: whether the straightforward linear transformation is expressive enough to fully cap- ture the rich information distributed in the ex- tracted partial-representations? In this work, we provide …

Content truncated to 500 words