On The Alignment Problem In Multi-Head Attention-Based Neural Machine Translation

Proceedings of the Third Conference on Machine Translation (WMT), Volume 1: Research Papers, pages 177–185 Belgium, Brussels, October 31 - Novermber 1, 2018. c⃝2018 Association for Computational Linguistics https://doi.org/10.18653/v1/W18-64018 On The Alignment Problem In Multi-Head Attention-Based Neural Machine Translation Tamer Alkhouli, Gabriel Bretschner, and Hermann Ney Human Language Technology and Pattern Recognition Group Computer Science Department RWTH Aachen University D-52056 Aachen, Germany <surname>@i6.informatik.rwth-aachen.de Abstract This work investigates the alignment prob- lem in state-of-the-art multi-head attention models based on the transformer architecture. We demonstrate that alignment extraction in transformer models can be improved by aug- menting an additional alignment head to the multi-head source-to-target attention compo- nent. This is used to compute sharper atten- tion weights. We describe how to use the alignment head to achieve competitive per- formance. To study the effect of adding the alignment head, we simulate a dictionary- guided translation task, where the user wants to guide translation using pre-defined dictio- nary entries. Using the proposed approach, we achieve up to 3.8% BLEU improvement when using the dictionary, in comparison to 2.4% BLEU in the baseline case. We also propose alignment pruning to speed up decod- ing in alignment-based neural machine trans- lation (ANMT), which speeds up translation by a factor of 1.8 without loss in translation performance. We carry out experiments on the shared WMT 2016 English→Romanian news task and the BOLT Chinese→English discus- sion forum task. 1 Introduction Attention-based neural machine translation (NMT) (Bahdanau et al., 2015) uses an atten- tion layer to determine which part of the input sequence to focus on during decoding. This com- ponent eliminates the need for explicit alignment modeling. In conventional phrase-based statistical machine translation (Koehn et al., 2003), word alignment is modeled explicitly, making it clear which word or phrase is being translated. The lack of explicit alignment use in attention-based models makes it harder to determine which target words are generated using which source words. While this is not necessarily needed for trans- lation itself, alignments can be useful in certain applications, e.g. when the customer wants to enforce specific translation of certain words. One simple solution is to use maximum atten- tion weights to extract the alignment, but this can result in wrong alignments in the case where the maximum attention weight is not pointing to the word being translated. Such cases are not uncom- mon in NMT, making the use of attention weights as alignment replacement non-trivial (Chatterjee et al., 2017; Hasler et al., 2018). Alignment ex- traction is even less clear for transformer models (Vaswani et al., 2017), which currently produce state-of-the-art results. These models use multiple attention components for each of the stacked de- coder layers. In this work we focus our study on these models since they usually outperform single- attention-head recurrent neural network (RNN) at- tention models.1 Alignment-based NMT (Alkhouli et al., 2016) uses neural models trained using explicit hard alignments to generate translation. These systems include explicit alignment modeling, making them more convenient for tasks where the source-to- target alignment …

Content truncated to 500 words