An Operation Sequence Model for Explainable Neural Machine Translation

Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 175–186 Brussels, Belgium, November 1, 2018. c⃝2018 Association for Computational Linguistics 175 An Operation Sequence Model for Explainable Neural Machine Translation Felix Stahlberg and Danielle Saunders and Bill Byrne Department of Engineering University of Cambridge, UK {fs439,ds636,wjb31}@cam.ac.uk Abstract We propose to achieve explainable neural ma- chine translation (NMT) by changing the out- put representation to explain itself. We present a novel approach to NMT which generates the target sentence by monotonically walking through the source sentence. Word reorder- ing is modeled by operations which allow set- ting markers in the target sentence and move a target-side write head between those mark- ers. In contrast to many modern neural mod- els, our system emits explicit word alignment information which is often crucial to practi- cal machine translation as it improves explain- ability. Our technique can outperform a plain text system in terms of BLEU score under the recent Transformer architecture on Japanese- English and Portuguese-English, and is within 0.5 BLEU difference on Spanish-English. 1 Introduction Neural machine translation (NMT) mod- els (Sutskever et al., 2014; Bahdanau et al., 2015; Gehring et al., 2017; Vaswani et al., 2017) are remarkably effective in modelling the distri- bution over target sentences conditioned on the source sentence, and yield superior translation performance compared to traditional statistical machine translation (SMT) on many language pairs. However, it is often difficult to extract a comprehensible explanation for the predictions of these models as information in the network is represented by real-valued vectors or matri- ces (Ding et al., 2017). In contrast, the translation process in SMT is ‘transparent’ as it can identify the source word which caused a target word through word alignment. Most NMT models do not use the concept of word alignment. It is tempting to interpret encoder-decoder attention matrices (Bahdanau et al., 2015) in neural models as (soft) alignments, but previous work has found that the attention weights in NMT are often erratic (Cheng et al., 2016) and differ significantly from traditional word alignments (Koehn and Knowles, 2017; Ghader and Monz, 2017). We will discuss the difference between attention and alignment in detail in Sec. 4. The goal of this paper is explainable NMT by developing a trans- parent translation process for neural models. Our approach does not change the neural architecture, but represents the translation together with its alignment as a linear sequence of operations. The neural model predicts this operation sequence, and thus simultaneously generates a translation and an explanation for it in terms of alignments from the target words to the source words that generate them. The operation sequence is “self- explanatory”; it does not explain an underlying NMT system but is rather a single representation produced by the NMT system that can be used to generate translations along with an accompanying explanatory alignment to the source sentence. We report competitive results of our method on Spanish-English, Portuguese-English, and Japanese-English, with the benefit of producing hard alignments …

Content truncated to 500 words