Leveraging Local and Global Patterns for Self-Attention Networks

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3069–3075 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 3069 Leveraging Local and Global Patterns for Self-Attention Networks Mingzhou Xu† Derek F. Wong†∗ Baosong Yang† Yue Zhang‡ Lidia S. Chao† †NLP2CT Lab / Department of Computer and Information Science, University of Macau ‡School of Engineering, Westlake University nlp2ct.{mzxu,baosong}@gmail.com, {derekfw,lidiasc}@um.edu.mo, yue.zhang@wias.org.cn Abstract Self-attention networks have received increas- ing research attention. By default, the hid- den states of each word are hierarchically cal- culated by attending to all words in the sen- tence, which assembles global information. However, several studies pointed out that tak- ing all signals into account may lead to over- looking neighboring information (e.g. phrase pattern). To address this argument, we pro- pose a hybrid attention mechanism to dynam- ically leverage both of the local and global in- formation. Specifically, our approach uses a gating scalar for integrating both sources of the information, which is also convenient for quantifying their contributions. Experiments on various neural machine translation tasks demonstrate the effectiveness of the proposed method. The extensive analyses verify that the two types of contexts are complementary to each other, and our method gives highly effec- tive improvements in their integration. 1 Introduction Self-attention networks (SANs) (Parikh et al., 2016; Lin et al., 2017) have shown promising re- sults for a range of NLP tasks, including machine translation (Vaswani et al., 2017), contextualized word embedding learning (Devlin et al., 2019), de- pendency parsing (Kitaev and Klein, 2018) and semantic role labeling (Tan et al., 2018). They learn hidden representations of a sequence by let- ting each word attend to all words in the sentence regardless of their distances. Such a fully con- nected structure endows SANs with the appealing strength of collecting the global information (Yu et al., 2018; Shen et al., 2018; Chen et al., 2018; Zhang et al., 2017a; Yang et al., 2019a). However, some recent researches observe that a fully connected SANs may overlook the important ∗Corresponding author neighboring information (Luong et al., 2015; Sper- ber et al., 2018; Yang et al., 2019a). They find that SANs can be empirically enhanced by restricting the attention scope to a local area. One interest- ing question arises: how the local and global pat- terns quantitatively affect the SANs. To this end, we make empirical investigations with a hybrid at- tention mechanism, which integrates a local and a global attentive representation via a gating scalar. Empirical results on English-to-German and Japanese-to-English tasks demonstrate the effec- tiveness of using both the local and global in- formation, which are shown complementary with each other. Our conceptually simple model con- sistently improves the performance over existing methods with fewer parameters. The probing tasks demonstrate that the local information is benefi- cial to the extraction of syntactic features, integrat- ing with the global information further improves the performance on semantic probing tasks. The quantification analysis of gating scalar also indi- cates that different types of …

Content truncated to 500 words