A Simple and Effective Unified Encoder for Document-Level Machine Translation

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3505–3511 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 3505 A Simple and Effective Unified Encoder for Document-Level Machine Translation Shuming Ma, Dongdong Zhang, Ming Zhou Microsoft Research Asia {shumma, dozhang, mingzhou}@microsoft.com Abstract Most of the existing models for document- level machine translation adopt dual-encoder structures. The representation of the source sentences and the document-level contexts1 are modeled with two separate encoders. Al- though these models can make use of the document-level contexts, they do not fully model the interaction between the contexts and the source sentences, and can not directly adapt to the recent pre-training models (e.g., BERT) which encodes multiple sentences with a single encoder. In this work, we propose a simple and effective unified encoder that can outperform the baseline models of dual- encoder models in terms of BLEU and ME- TEOR scores. Moreover, the pre-training mod- els can further boost the performance of our proposed model. 1 Introduction Thanks to the development of the deep learning methods, the machine translation systems have achieved good performance that is even comparable with human translation in the news domain (Hassan et al., 2018). However, there are still some prob- lems with machine translation in the document- level context (L¨aubli et al., 2018). Therefore, more recent work (Jean et al., 2017; Wang et al., 2017; Tiedemann and Scherrer, 2017; Maruf and Haffari, 2018; Bawden et al., 2018; Voita et al., 2019a; Junczys-Dowmunt, 2019) is focusing on the document-level machine translation. Most of the existing models (Zhang et al., 2018; Maruf et al., 2019; Werlen et al., 2018) for document-level machine translation use two encoders to model the source sentences and the document-level contexts. Figure 1a illustrates the structure of these models. They extend the standard 1In this work, document-level contexts denote the sur- rounding sentences of the current source sentence. Transformer Encoder Transformer Encoder Transformer Decoder Translation Context Source (a) Dual-Encoder Structure Transformer Encoder Transformer Decoder Translation Context Source + (b) Uni-Encoder Structure Figure 1: The overview of the dual-encoder structure and the uni-encoder structure for document-level ma- chine translation. Transformer model with a new context encoder, and the encoder for source sentences is conditioned on this context encoder. However, they do not fully model the interaction between the contexts and the source sentences because the self-attention lay- ers are performed inside each encoder separately. Moreover, it cannot be directly adapted to the re- cent pre-training models (Devlin et al., 2019; Peters et al., 2018; Radford et al., 2019; Dong et al., 2019; Song et al., 2019; Lample and Conneau, 2019), which encodes multiple sentences with a single encoder. Different from the dual-encoder structure, the uni-encoder structure takes the concatenation of contexts and source sentences as the input (as shown in Figure 1b). Therefore, when modeling the contexts, it can make full use of the interac- tion between the source sentences and the contexts, while the dual-encoder model fails to exploit this information. Moreover, the …

Content truncated to 500 words