Context-aware Decoder for Neural Machine Translation using a Target-side Document-Level Language Model

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5781–5791 June 6–11, 2021. ©2021 Association for Computational Linguistics 5781 Context-aware Decoder for Neural Machine Translation using a Target-side Document-Level Language Model Amane Sugiyama The University of Tokyo∗ sugi@tkl.iis.u-tokyo.ac.jp Naoki Yoshinaga Institute of Industrial Science, The University of Tokyo ynaga@iis.u-tokyo.ac.jp Abstract Although many end-to-end context-aware neu- ral machine translation models have been pro- posed to incorporate inter-sentential contexts in translation, these models can be trained only in domains where parallel documents with sen- tential alignments exist. We therefore present a simple method to perform context-aware decoding with any pre-trained sentence-level translation model by using a document-level language model. Our context-aware decoder is built upon sentence-level parallel data and target-side document-level monolingual data. From a theoretical viewpoint, our core contri- bution is the novel representation of contex- tual information using point-wise mutual in- formation between context and the current sen- tence. We demonstrate the effectiveness of our method on English to Russian translation, by evaluating with BLEU and contrastive tests for context-aware translation. 1 Introduction Neural machine translation (NMT) has typically been explored in sentence-level translation settings. Such sentence-level NMT models inevitably suf- fer from ambiguities when a source sentence has multiple plausible interpretations. Examples of such ambiguities include anaphora, ellipsis, and lexical coherence (Voita et al., 2019b); although resolving these ambiguities has only a minor im- pact on the translation performance measured by BLEU scores (Papineni et al., 2002), they are vital in smoothly reading the translated documents. To address this issue, context-aware NMT models which incorporate document-level information in translation have recently been explored (Jean et al., 2017; Wang et al., 2017; Tiedemann and Scherrer, 2017; Maruf and Haffari, 2018; Voita et al., 2018; Bawden et al., 2018; Miculicich et al., 2018; Maruf et al., 2019; Voita et al., 2019b; Yu et al., 2020; ∗Currently at Mitsubishi UFJ Morgan Stanley Securities Kang et al., 2020; Zhang et al., 2020). Most of these models are end-to-end models that require document-level parallel data with sentential align- ments for training. However, this data is available in only a few domains (Sugiyama and Yoshinaga, 2019). Researchers have therefore started to utilize target-side monolingual data to construct auxiliary models which help a sentence-level NMT model per- form context-aware translation (Voita et al., 2019a; Stahlberg et al., 2019; Yu et al., 2020). In this study, we propose a simple yet effective approach to context-aware NMT using two primi- tive components, a sentence-level NMT model and a document-level language model (LM). We can independently train the two components on com- mon sentence-level parallel data and document- level monolingual data, respectively, without us- ing document-level parallel data. Our approach thereby makes it possible to perform context-aware translation with any pre-trained sentence-level NMT model, using a pre-trained document-level LM. To give a probabilistic foundation to this combi- nation of two independent models, we exploit the probabilistic nature of NMT decoding. When gener- ating a sequence, a left-to-right decoder …

Content truncated to 500 words