Breaking the Corpus Bottleneck for Context-Aware Neural Machine Translation with Cross-Task Pre-training

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 2851–2861 August 1–6, 2021. ©2021 Association for Computational Linguistics 2851 Breaking Corpus Bottleneck for Context-Aware Neural Machine Translation with Cross-Task Pre-training 1Linqing Chen 1Junhui Li∗ 1Zhengxian Gong 2Boxing Chen 2Weihua Luo 1Min Zhang 1Guodong Zhou 1School of Computer Science and Technology, Soochow University, Suzhou, China 2Alibaba DAMO Academy {lijunhui, zhxgong, minzhang, gdzhou}@suda.edu.cn, lqchen21@gmail.com, {boxing.cbx, weihua.luowh}@alibaba-inc.com Abstract Context-aware neural machine translation (NMT) remains challenging due to the lack of large-scale document-level parallel dataset. To break the corpus bottleneck, in this pa- per we aim to improve context-aware NMT by taking the advantage of the availability of both large-scale sentence-level parallel dataset and source-side monolingual documents.1 To this end, we propose two pre-training tasks. One learns to translate a sentence from source language to target language on the sentence- level parallel dataset while the other learns to translate a document from deliberately noised to original on the monolingual docu- ments. Importantly, the two pre-training tasks are jointly and simultaneously learned via the same model, thereafter fine-tuned on scale- limited parallel documents from both sentence- level and document-level perspectives. Exper- imental results on four translation tasks show that our approach significantly improves trans- lation performance. One nice property of our approach is that the fine-tuned model can be used to translate both sentences and docu- ments. 1 Introduction Document-level context-aware neural machine translation (NMT) aims to translate sentences in a document under the guidance of document-level context. Recent years have witnessed great im- provement in context-aware NMT with extensive attempts at effectively leveraging document-level context ((Tiedemann and Scherrer, 2017; Maruf and Haffari, 2018; Maruf et al., 2019), to name a few). However, the performance of context- aware NMT still suffers from the size of paral- lel document dataset. On the one hand, unlike ∗Corresponding Author: Junhui Li. 1If not specified, monolingual documents are all for source- side through this paper. sentence-level translation models which could be well trained on large-scale sentence-level parallel datasets, the translation models of context-aware NMT may result in insufficient training. On the other hand, with only scale-limited source-side documents, the context encoders may fail to ef- fectively extract useful context from the whole doc- ument.2 On the contrary, large-scale of parallel sentence corpora, and especially monolingual doc- ument corpora are much easier to find. In this paper, our goal is to break the corpus bottleneck for context-aware NMT by leveraging both large- scale sentence-level parallel dataset and monolin- gual documents. Specifically, we aim to use the former to boost the performance of translation mod- els while employ the latter to enhance the context encoders’ capability of capturing useful context information. There have been several attempts to boost context-aware NMT performance in the scenar- ios where the document-level parallel dataset is scale-limited, or even not available. On the one hand, sentence-level parallel dataset is a natural re- source to use. For example, Zhang et al. (2018) pro- …

Content truncated to 500 words