Saliency-driven Word Alignment Interpretation for Neural Machine Translation

Proceedings of the Fourth Conference on Machine Translation (WMT), Volume 1: Research Papers, pages 1–12 Florence, Italy, August 1-2, 2019. c⃝2019 Association for Computational Linguistics 1 Saliency-driven Word Alignment Interpretation for Neural Machine Translation Shuoyang Ding Hainan Xu Philipp Koehn Center for Language and Speech Processing Johns Hopkins University {dings, hxu31, phi}@jhu.edu Abstract Despite their original goal to jointly learn to align and translate, Neural Machine Transla- tion (NMT) models, especially Transformer, are often perceived as not learning inter- pretable word alignments. In this paper, we show that NMT models do learn interpretable word alignments, which could only be re- vealed with proper interpretation methods. We propose a series of such methods that are model-agnostic, are able to be applied either offline or online, and do not require parame- ter update or architectural change. We show that under the force decoding setup, the align- ments induced by our interpretation method are of better quality than fast-align for some systems, and when performing free decoding, they agree well with the alignments induced by automatic alignment tools. 1 Introduction Neural Machine Translation (NMT) has made lots of advancements since its inception. One of the key innovations that led to the largest improve- ments is the introduction of the attention mecha- nism (Bahdanau et al., 2014; Luong et al., 2015), which jointly learns word alignment and transla- tion. Since then, the attention mechanism has gradually become a general technique in various NLP tasks, including summarization (Rush et al., 2015; See et al., 2017), natural language infer- ence (Parikh et al., 2016) and speech recognition (Chorowski et al., 2015; Chan et al., 2016). Although word alignment is no longer a integral step like the case for Statistical Machine Transla- tion (SMT) systems (Brown et al., 1993; Koehn et al., 2003), there is a resurgence of interest in the community to study word alignment for NMT models. Even for NMT, word alignments are use- ful for error analysis, inserting external vocabular- ies, and providing guidance for human translators wir glauben nicht , daß wir nur rosinen herauspicken sollten . we do not believe that we should cherry-pick . wir glauben nicht , daß wir nur rosinen herauspicken sollten . we do not believe that we should cherry-pick . (a) reference (b) attention wir glauben nicht , daß wir nur rosinen herauspicken sollten . we do not believe that we should cherry-pick . wir glauben nicht , daß wir nur rosinen herauspicken sollten . we do not believe that we should cherry-pick . (c) word saliency (d) word saliency with SmoothGrad Figure 1: Comparison of our saliency-based word alignment interpretation of convolutional NMT model with reference and attention interpretation. in computer-aided translation. When aiming for the most accurate alignments, the state-of-the-art tools include GIZA++ (Brown et al., 1993; Och and Ney, 2003) and fast-align (Dyer et al., 2013), which are all external models invented in SMT era and need to be run as a separate post-processing step after the full sentence translation is complete. As a …

Content truncated to 500 words