Multi-Granularity Self-Attention for Neural Machine Translation

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 887–897, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 887 Multi-Granularity Self-Attention for Neural Machine Translation Jie Hao∗ Florida State University haoj8711@gmail.com Xing Wang Tencent AI Lab brightxwang@tencent.com Shuming Shi Tencent AI Lab shumingshi@tencent.com Jinfeng Zhang Florida State University jinfeng@stat.fsu.edu Zhaopeng Tu Tencent AI Lab zptu@tencent.com Abstract Current state-of-the-art neural machine trans- lation (NMT) uses a deep multi-head self- attention network with no explicit phrase in- formation. However, prior work on statis- tical machine translation has shown that ex- tending the basic translation unit from words to phrases has produced substantial improve- ments, suggesting the possibility of improv- ing NMT performance from explicit model- ing of phrases. In this work, we present multi-granularity self-attention (MG-SA): a neural network that combines multi-head self- attention and phrase modeling. Specifically, we train several attention heads to attend to phrases in either n-gram or syntactic for- malism. Moreover, we exploit interactions among phrases to enhance the strength of structure modeling – a commonly-cited weak- ness of self-attention. Experimental results on WMT14 English-to-German and NIST Chinese-to-English translation tasks show the proposed approach consistently improves per- formance. Targeted linguistic analysis reveals that MG-SA indeed captures useful phrase in- formation at various levels of granularities. 1 Introduction Recently, TRANSFORMER (Vaswani et al., 2017), implemented as deep multi-head self-attention networks (SANs), has become the state-of-the-art neural machine translation (NMT) model in recent years. The popularity of SANs lies in its high par- allelization in computation, and flexibility in mod- eling dependencies regardless of distance by ex- plicitly attending to all the signals. More recently, an in-depth study (Raganato and Tiedemann, 2018) reveals that SANs generally focus on disperse words and ignore continuous phrase patterns, which have proven essential in both statistical machine translation (SMT, Koehn ∗Work done when interning at Tencent AI Lab. et al., 2003; Chiang, 2005; Liu et al., 2006) and NMT (Eriguchi et al., 2016; Wang et al., 2017; Yang et al., 2018; Zhao et al., 2018). To alleviate this problem, in this work we pro- pose multi-granularity self-attention (MG-SA), which offers SANs the ability to model phrases and meanwhile maintain their simplicity and flex- ibility. The starting point for our approach is an observation: the power of multiple heads in SANs is not fully exploited. For example, Li et al. (2018) show that different attention heads generally at- tend to the same positions, and Voita et al. (2019) reveal that only specialized attention heads do the heavy lifting while the rest can be pruned. Accord- ingly, we spare several attention heads for model- ing phrase patterns for SANs. Specifically, we use two representative types of phrases that are widely-used in SMT models: n-gram phrases (Koehn et al., 2003) to use sur- face of adjacent words, and syntactic phrases (Liu et al., 2006) induced from syntactic trees to rep- resent well-formed structural information. We first partition the input sentence …

Content truncated to 500 words