Jointly Learning to Align and Translate with Transformer Models

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 4453–4462, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 4453 Jointly Learning to Align and Translate with Transformer Models Sarthak Garg Stephan Peitz Udhyakumar Nallasamy Matthias Paulik Apple Inc. {sarthak garg, speitz, udhay, mpaulik}@apple.com Abstract The state of the art in machine translation (MT) is governed by neural approaches, which typically provide superior translation accuracy over statistical approaches. However, on the closely related task of word alignment, tradi- tional statistical word alignment models of- ten remain the go-to solution. In this pa- per, we present an approach to train a Trans- former model to produce both accurate trans- lations and alignments. We extract discrete alignments from the attention probabilities learnt during regular neural machine trans- lation model training and leverage them in a multi-task framework to optimize towards translation and alignment objectives. We demonstrate that our approach produces com- petitive results compared to GIZA++ trained IBM alignment models without sacrificing translation accuracy and outperforms previous attempts on Transformer model based word alignment. Finally, by incorporating IBM model alignments into our multi-task training, we report significantly better alignment accu- racies compared to GIZA++ on three publicly available data sets. Our implementation has been open-sourced1. 1 Introduction Neural machine translation (NMT) constitutes the state of the art in MT, with the Transformer model architecture (Vaswani et al., 2017) beating other neural architectures in competitive MT evalua- tions. The attention mechanism used in NMT models was motivated by the need to model word alignments, however it is now well known that the attention probabilities can differ significantly from word alignments in the traditional sense (Koehn and Knowles, 2017), since attending to the con- text words rather than the aligned source words 1Code can be found at https://github.com/ pytorch/fairseq/pull/1095 might be helpful for translation. The presence of multi-layer, multi-head attention mechanisms in the Transformer model further complicate inter- preting the attention probabilities and extracting high quality discrete alignments from them. Finding source to target word alignments has many applications in the context of MT. A straightforward application of word alignments is to generate bilingual lexica from parallel corpora. Word alignments have also been used for external dictionary assisted translation (Chatterjee et al., 2017; Alkhouli et al., 2018; Arthur et al., 2016) to improve translation of low frequency words or to comply with certain terminology guidelines. Doc- uments and webpages often contain word anno- tations such as formatting styles and hyperlinks, which need to be preserved in the translation. In such cases, word alignments can be used to trans- fer these annotations from the source sentence to its translation. In user facing translation services, providing word alignments as additional informa- tion to the users might improve their trust and con- fidence, and also help them to diagnose problems such as under-translation (Tu et al., 2016). In this work, we introduce an approach that teaches Transformer models to produce transla- …

Content truncated to 500 words