Using Whole Document Context in Neural Machine Translation

Using Whole Document Context in Neural Machine Translation Valentin Mac´e, Christophe Servan QWANT RESEARCH - 7 Rue Spontini, 75116 Paris, France initial.lastname@qwant.com Abstract In Machine Translation, considering the document as a whole can help to resolve ambiguities and inconsistencies. In this paper, we propose a simple yet promising approach to add contextual information in Neural Machine Transla- tion. We present a method to add source context that cap- ture the whole document with accurate boundaries, taking every word into account. We provide this additional infor- mation to a Transformer model and study the impact of our method on three language pairs. The proposed approach obtains promising results in the English-German, English- French and French-English document-level translation tasks. We observe interesting cross-sentential behaviors where the model learns to use document-level information to improve translation coherence. 1. Introduction Neural machine translation (NMT) has grown rapidly in the past years [1, 2]. It usually takes the form of an encoder- decoder neural network architecture in which source sen- tences are summarized into a vector representation by the encoder and are then decoded into target sentences by the decoder. NMT has outperformed conventional statistical ma- chine translation (SMT) by a significant margin over the past years, benefiting from gating and attention techniques. Vari- ous models have been proposed based on different architec- tures such as RNN [1], CNN [3] and Transformer [2], the lat- ter having achieved state-of-the-art performances while sig- nificantly reducing training time. However, by considering sentence pairs separately and ignoring broader context, these models suffer from the lack of valuable contextual information, sometimes leading to in- consistency in a translated document. Adding document- level context helps to improve translation of context-dependent parts. Previous study [4] showed that such context gives sub- stantial improvement in the handling of discourse phenom- ena like lexical disambiguation or co-reference resolution. Most document-level NMT approaches focus on adding contextual information by taking into account a set of sen- tences surrounding the current pair [5, 6, 7, 8, 9, 10]. While giving significant improvement over the context-agnostic ver- sions, none of these studies consider the whole document with well delimited boundaries. The majority of these ap- proaches also rely on structural modification of the NMT model [7, 8, 9, 10]. To the best of our knowledge, there is no existing work considering whole documents without struc- tural modifications. Contribution: We propose a preliminary study of a generic approach allowing any model to benefit from document-level information while translating sentence pairs. The core idea is to augment source data by adding document information to each sentence of a source corpus. This document infor- mation corresponds to the belonging document of a sentence and is computed prior to training, it takes every document word into account. Our approach focuses on pre-processing and consider whole documents as long as they have defined boundaries. We conduct experiments using the Transformer base model [2]. For the English-German language pair we use the full WMT 2019 parallel dataset. For the English- French language …

Content truncated to 500 words