Context-Interactive Pre-Training for Document Machine Translation

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3589–3595 June 6–11, 2021. ©2021 Association for Computational Linguistics 3589 Context-Interactive Pre-Training for Document Machine Translation Pengcheng Yang, Pei Zhang, Boxing Chen, Jun Xie, Weihua Luo Machine Intelligence Technology Lab Alibaba Group Hangzhou, China {mingyang.ypc, xiaoyi.zp, boxing.cbx, qingjing.xj, weihua.luowh}@alibaba-inc.com Abstract Document machine translation aims to trans- late the source sentence into the target lan- guage in the presence of additional contex- tual information. However, it typically suffers from a lack of doc-level bilingual data. To rem- edy this, here we propose a simple yet effec- tive context-interactive pre-training approach, which targets benefiting from external large- scale corpora. The proposed model performs inter sentence generation to capture the cross- sentence dependency within the target docu- ment, and cross sentence translation to make better use of valuable contextual information. Comprehensive experiments illustrate that our approach can achieve state-of-the-art perfor- mance on three benchmark datasets, which sig- nificantly outperforms a variety of baselines. 1 Introduction Document machine translation (Doc-MT) aims at utilizing the surrounding contexts of the source sen- tence to tackle some linguistic consistency prob- lems (e.g., deixis, ellipsis, and lexical cohesion) in translation (Tiedemann and Scherrer, 2017). How- ever, due to the introduction of extra contexts, it also presents several intractable challenges: (1) Data scarcity of document-level bilingual corpora. Since most bilingual corpora are pre- served by sentence, well-aligned document-level data is relatively scarce (Zhang et al., 2018), espe- cially for low-resource languages or domains. Such a data sparsity not only impairs the effective train- ing of neural machine translation (NMT) models, but also tends to result in potential overfitting. (2) Effective utilization of valuable information contained in extra contexts. Although some ef- forts (Wang et al., 2017; Tu et al., 2018) have strived to incorporate contextual information via various architectures, they only observe minor per- formance gains compared with traditional sentence machine translation (Sent-MT). Recent work (Li et al., 2020) also reveals that contextual informa- tion cannot be fully leveraged by some existing approaches, where the source contexts tend to act as the data noise enriching the training signals. (3) Modeling of cross-sentence dependency within the target document. Since the input of Doc- MT focuses on documents consisting of multiple sentences, the decoder should be able to deal with some discourse phenomena like coreference res- olution, lexical cohesion, and lexical disambigua- tion. (Voita et al., 2019b). This goal requires the modeling of cross-sentence dependency within the target document. To tackle the above three challenges, here we propose a simple yet effective context-interactive pre-training approach for Doc-MT. The proposal consists of three pre-training tasks, whose sketch is presented in Figure 1. Specifically, the cross sen- tence translation task (CST in Figure 1 (A)) strives to generate the target sentence in the absence of the source sentence and only based on the source con- texts. With such a goal, the model is encouraged to maximize the utilization of extra contexts. To cap- …

Content truncated to 500 words