End-to-End Neural Word Alignment Outperforms GIZA++

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1605–1617 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 1605 End-to-End Neural Word Alignment Outperforms GIZA++ Thomas Zenkel, Joern Wuebker, John DeNero Lilt, Inc. first name@lilt.com Abstract Word alignment was once a core unsuper- vised learning task in natural language pro- cessing because of its essential role in train- ing statistical machine translation (MT) mod- els. Although unnecessary for training neu- ral MT models, word alignment still plays an important role in interactive applications of neural machine translation, such as annotation transfer and lexicon injection. While statisti- cal MT methods have been replaced by neu- ral approaches with superior performance, the twenty-year-old GIZA++ toolkit remains a key component of state-of-the-art word alignment systems. Prior work on neural word alignment has only been able to outperform GIZA++ by using its output during training. We present the first end-to-end neural word alignment method that consistently outperforms GIZA++ on three data sets. Our approach repurposes a Transformer model trained for supervised translation to also serve as an unsupervised word alignment model in a manner that is tightly integrated and does not affect transla- tion quality. 1 Introduction Although word alignments are no longer necessary to train machine translation (MT) systems, they still play an important role in applications of neu- ral MT. For example, they enable injection of an external lexicon into the inference process to en- force the use of domain-specific terminology or improve the translations of low-frequency content words (Arthur et al., 2016). The most important application today for word alignments is to trans- fer text annotations from source to target (M¨uller, 2017; Tezcan and Vandeghinste, 2011; Joanis et al., 2013; Escartın and Arcedillo, 2015). For exam- ple, if part of a source sentence is underlined, the corresponding part of its translation should be un- derlined as well. HTML tags and other markup must be transferred for published documents. Al- though annotations could in principle be generated directly as part of the output sequence, they are instead typically transferred via word alignments because example annotations typically do not exist in MT training data. The Transformer architecture provides state-of- the-art performance for neural machine translation (Vaswani et al., 2017). The decoder has multiple layers, each with several attention heads, which makes it difficult to interpret attention activations as word alignments. As a result, the most widely used tools to infer word alignments, namely GIZA++ (Och and Ney, 2003) and FastAlign (Dyer et al., 2013), are still based on the statistical IBM word alignment models developed nearly thirty years ago (Brown et al., 1993). No previous unsupervised neural approach has matched their performance. Recent work on alignment components that are integrated into neural translation models either un- derperform the IBM models or must use the output of IBM models during training to outperform them (Zenkel et al., 2019; Garg et al., 2019). This work combines key components from Zenkel et al. (2019) and Garg et al. (2019) and …

Content truncated to 500 words