G-Transformer for Document-Level Machine Translation

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 3442–3455 August 1–6, 2021. ©2021 Association for Computational Linguistics 3442 G-Transformer for Document-level Machine Translation Guangsheng Bao1,2, Yue Zhang∗,1,2, Zhiyang Teng1,2, Boxing Chen3 and Weihua Luo3 1 School of Engineering, Westlake University 2 Institute of Advanced Technology, Westlake Institute for Advanced Study 3 DAMO Academy, Alibaba Group Inc. {baoguangsheng, zhangyue, tengzhiyang}@westlake.edu.cn {boxing.cbx, weihua.luowh}@alibaba-inc.com Abstract Document-level MT models are still far from satisfactory. Existing work extend translation unit from single sentence to multiple sentences. However, study shows that when we further en- large the translation unit to a whole document, supervised training of Transformer can fail. In this paper, we find such failure is not caused by overfitting, but by sticking around local min- ima during training. Our analysis shows that the increased complexity of target-to-source at- tention is a reason for the failure. As a solution, we propose G-Transformer, introducing local- ity assumption as an inductive bias into Trans- former, reducing the hypothesis space of the attention from target to source. Experiments show that G-Transformer converges faster and more stably than Transformer, achieving new state-of-the-art BLEU scores for both non- pretraining and pre-training settings on three benchmark datasets. 1 Introduction Document-level machine translation (MT) has re- ceived increasing research attention (Gong et al., 2011; Hardmeier et al., 2013; Garcia et al., 2015; Miculicich et al., 2018a; Maruf et al., 2019; Liu et al., 2020). It is a more practically useful task compared to sentence-level MT because typical in- puts in MT applications are text documents rather than individual sentences. A salient difference be- tween document-level MT and sentence-level MT is that for the former, much larger inter-sentential context should be considered when translating each sentence, which include discourse structures such as anaphora, lexical cohesion, etc. Studies show that human translators consider such contexts when conducting document translation (Hardmeier, 2014; L¨aubli et al., 2018). Despite that neural mod- els achieve competitive performances on sentence- ∗* Corresponding author. Context Encoder Source Encoder Target Decoder Source Translation Context (a) Sentence-by-sentence Translation Source Encoder Target Decoder Source1 Translation1 Source2 … Translation2 … (b) Multi-sentence Translation Source1 Translation1 Source2 … Translation2 … … … ① ② ③ ① ② ③ (c) G-Transformer (Doc-by-doc Translation) Figure 1: Overview of model structures for document- level machine translation. level MT, the performance of document-level MT is still far from satisfactory. Existing methods can be mainly classified into two categories. The first category translates a doc- ument sentence by sentence using a sequence-to- sequence neural model (Zhang et al., 2018; Mi- culicich et al., 2018b; Maruf et al., 2019; Zheng et al., 2020). Document-level context is integrated into sentence-translation by introducing additional context encoder. The structure of such a model is shown in Figure 1(a). These methods suffer from two limitations. First, the context needs to be encoded separately for translating each sentence, which adds to the runtime complexity. Second, more importantly, information exchange cannot be …

Content truncated to 500 words