On the Word Alignment from Neural Machine Translation

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1293–1303 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 1293 On the Word Alignment from Neural Machine Translation∗ Xintong Li1, Guanlin Li2, Lemao Liu3, Max Meng1, Shuming Shi3 1The Chinese University of Hong Kong 2Harbin Institute of Technology 3Tencent AI Lab {znculee, epsilonlee.green}@gmail.com {redmondliu, shumingshi}@tencent.com max.meng@cuhk.edu.hk Abstract Prior researches suggest that neural machine translation (NMT) captures word alignment through its attention mechanism, however, this paper finds attention may almost fail to cap- ture word alignment for some NMT models. This paper thereby proposes two methods to induce word alignment which are general and agnostic to specific NMT models. Experi- ments show that both methods induce much better word alignment than attention. This pa- per further visualizes the translation through the word alignment induced by NMT. In par- ticular, it analyzes the effect of alignment er- rors on translation errors at word level and its quantitative analysis over many testing exam- ples consistently demonstrate that alignment errors are likely to lead to translation errors measured by different metrics. 1 Introduction Machine translation aims at modeling the seman- tic equivalence between a pair of source and target sentences (Koehn, 2009), and word alignment tries to model the semantic equivalence between a pair of source and target words (Och and Ney, 2003). As a sentence consists of words, word alignment is conceptually related to machine translation and such a relation can be traced back to the birth of statistical machine translation (SMT) (Brown et al., 1993), where word alignment is the basis of SMT models and its accuracy is generally helpful to improve translation quality (Koehn et al., 2003; Liu et al., 2005). In neural machine translation (NMT), it is also important to study word alignment, because word alignment provides natural ways to understanding black-box NMT models and analyzing their trans- lation errors (Ding et al., 2017). Prior researches ∗Work done while X. Li interning at Tencent AI Lab. L. Liu is the corresponding author. observed that word alignment is captured by NMT through attention for recurrent neural network based NMT with a single attention layer (Bah- danau et al., 2014; Mi et al., 2016; Liu et al., 2016; Li et al., 2018). Unfortunately, we surpris- ingly find that attention may almost fail to capture word alignment for NMT models with multiple at- tentional layers such as TRANSFORMER (Vaswani et al., 2017), as demonstrated in our experiments. In this paper, we propose two methods to in- duce word alignment from general NMT models and answer a fundamental question that how much word alignment NMT models can learn (§ 3). The first method explicitly builds a word align- ment model between a pair of source and tar- get word representations encoded by NMT mod- els, and then it learns additional parameters for this word alignment model with the supervision from an external aligner similar to Mi et al. (2016) and Liu et al. (2016). The second …

Content truncated to 500 words