Context-aware Neural Machine Translation with Mini-batch Embedding

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 2513–2521 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 2513 Context-aware Neural Machine Translation with Mini-batch Embedding Makoto Morishita1,2, Jun Suzuki2, Tomoharu Iwata1, Masaaki Nagata1 NTT Communication Science Laboratories, NTT Corporation1 Tohoku University2 {makoto.morishita.gr, tomoharu.iwata.gy, masaaki.nagata.et}@hco.ntt.co.jp jun.suzuki@ecei.tohoku.ac.jp Abstract It is crucial to provide an inter-sentence con- text in Neural Machine Translation (NMT) models for higher-quality translation. With the aim of using a simple approach to incorporate inter-sentence information, we propose mini- batch embedding (MBE) as a way to represent the features of sentences in a mini-batch. We construct a mini-batch by choosing sentences from the same document, and thus the MBE is expected to have contextual information across sentences. Here, we incorporate MBE in an NMT model, and our experiments show that the proposed method consistently outperforms the translation capabilities of strong baselines and improves writing style or terminology to fit the document’s context.1 1 Introduction Current standard neural machine translation (NMT) models (Sutskever et al., 2014; Bahdanau et al., 2015; Luong et al., 2015; Vaswani et al., 2017) translate sentences in a sentence-by-sentence man- ner. However, some have argued that it is critical to consider the inter-sentence context in handling discourse phenomena (Hardmeier, 2012), which include coherence, cohesion, coreference (Baw- den et al., 2018; Nagata and Morishita, 2020), and writing style (Yamagishi et al., 2016). To correctly translate these linguistic features, some works provide additional context information to an NMT model by concatenating the previous sen- tence (Tiedemann and Scherrer, 2017), applying a context encoder (Bawden et al., 2018; Miculicich et al., 2018; Voita et al., 2018), or using a cache- based network (Tu et al., 2018; Kuang et al., 2018). Most of the previous studies have considered only a few previous context sentences. Several 1Our implementation is publicly available: https:// github.com/nttcslab-nlp/mbe-nmt methods, such as the cache-based network, con- sider long-range context but heavily modify the standard NMT models and require additional train- ing/decoding steps. Our goal is to make a simple but effective context-aware NMT model, which does not require heavy modification to standard NMT models and can handle a wider inter-sentence context. To this end, we propose a method to create an embedding that represents the contextual infor- mation of a document. To create this embedding, we focused on the mini-batch, which is commonly used in NMT training and decoding for efficient GPU computation. We modified the mini-batch cre- ation algorithm to choose sentences from a single document and created an embedding that repre- sents the features of the mini-batch. We call this embedding mini-batch embedding (MBE) and in- corporate it in the NMT model to exploit contextual information across the sentences in the mini-batch. Our main contributions can be summarized as follows: (i) We introduce mini-batch embedding to represent the features of sentences in a mini- batch. (ii) We incorporate mini-batch embedding in NMT to achieve simple context-aware translation and find that our approach improves translation performance by …

Content truncated to 500 words