Investigation of Transformer-based Latent Attention Models for Neural Machine Translation

Investigation of Transformer-based Latent Attention Models for Neural Machine Translation Parnia Bahar1,2 bahar@cs.rwth-aachen.de Nikita Makarov1 nikita.makarov@rwth-aachen.de Hermann Ney1,2 ney@cs.rwth-aachen.de 1Human Language Technology and Pattern Recognition Group, Computer Science Department, RWTH Aachen University, 52074 Aachen, Germany 2AppTek GmbH, 52062 Aachen, Germany Abstract Current neural translation networks are based on an effective attention mechanism that can be considered as an implicit probabilistic notion of alignment. Such architectures do not guarantee a high quality alignment, even though alignments can easily be used for explainable machine translation. This work describes a latent variable attention model using the transformer architecture, where we carry out an approximate marginalization over alignments. We show that the alignment quality in transformer models can be improved by introducing a latent variable for the alignments. To study the effect of the latent model, we quantitatively and qualitatively analyze the extracted alignments from the multi-head attention. We demonstrate that this method slightly improves translation quality on four WMT 2018 shared translation tasks, as well as generating more focused alignments for better interpretability. 1 Introduction Current state-of-the-art neural machine translation (NMT) systems are based on attention models (Bahdanau et al., 2015; Luong et al., 2015; Gehring et al., 2017; Vaswani et al., 2017), where “soft attention” is used to focus on the most relevant parts of the source sequence while generating target words. This distribution can be considered as an implicit probabilistic notion of alignment as an intermediate step of the translation model. However, it does not work in the same way as its analogous alignment in conventional statistical machine translation (SMT) (Koehn and Knowles, 2017). Contrary to SMT, where a “hard alignment” is a transparent process that defines the correspondence between source and target words, it is often challenging for the fuzzy attention mechanism to extract a comprehensible alignment. Although the context vector, a weighted sum of the input states, includes the alignment information implicitly, the alignment model does not directly influence the final translation probability of a sentence. Hence, it is often hard to determine the contribution of the attention on the output. In this work, • we incorporate the attention model into the direct hidden Markov model (HMM) formulation in the transformer architecture such that the translation posterior distribution has a direct dependency on the source positions. • as to the interest in alignment based approaches, we intend to investigate whether the latent-based models are able to tackle the explainability problem of alignments especially Proceedings of the 14th Conference of the Association for Machine Translation in the Americas October 6 - 9, 2020, Volume 1: MT Research Track Page 7 for the transformer model. Similar to how some approaches in explainable NMT use the attention (Stahlberg et al., 2018; Zenkel et al., 2019; Garg et al., 2019), we also incorporate it as a key interpretation component into our work. From the statistical perspective, we revisit the formulation of the posterior probability of a given sentence by computing a separate translation score for every target-source word pair. Since marginalization is exponential …

Content truncated to 500 words