Multilingual Document-Level Translation Enables Zero-Shot Transfer From Sentences to Documents

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 4176 - 4192 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Multilingual Document-Level Translation Enables Zero-Shot Transfer From Sentences to Documents Biao Zhang1∗, Ankur Bapna2, Melvin Johnson2, Ali Dabirmoghaddam2, Naveen Arivazhagan2, Orhan Firat2 1 School of Informatics, University of Edinburgh 2 Google Research b.zhang@ed.ac.uk, {ankurbpn, melvinp, dabir, navari, orhanf}@google.com Abstract Document-level neural machine translation (DocNMT) achieves coherent translations by incorporating cross-sentence context. How- ever, for most language pairs there’s a short- age of parallel documents, although parallel sentences are readily available. In this paper, we study whether and how contextual model- ing in DocNMT is transferable via multilin- gual modeling. We focus on the scenario of zero-shot transfer from teacher languages with document level data to student languages with no documents but sentence level data, and for the first time treat document-level translation as a transfer learning problem. Using simple concatenation-based DocNMT, we explore the effect of 3 factors on the transfer: the num- ber of teacher languages with document level data, the balance between document and sen- tence level data at training, and the data condi- tion of parallel documents (genuine vs. back- translated). Our experiments on Europarl-7 and IWSLT-10 show the feasibility of multi- lingual transfer for DocNMT, particularly on document-specific metrics. We observe that more teacher languages and adequate data bal- ance both contribute to better transfer quality. Surprisingly, the transfer is less sensitive to the data condition, where multilingual DocNMT delivers decent performance with either back- translated or genuine document pairs. 1 Introduction Recent years have witnessed a trend moving from sentence-level neural machine translation (Sen- NMT) to its document-level counterpart (Doc- NMT). SenNMT inevitably suffers from translation errors related with document phenomena (Maruf et al., 2021) and delivers obviously inferior perfor- mance when compared against human translations and evaluated at a document level (Läubli et al., ∗Work done while Biao Zhang was interning at Google Research. w/ sentence pairs w/ sentence pairs w/ document pairs w/o document pairs Document-Rich Languages Document-Poor Languages SenNMT DocNMT zero-shot transfer Figure 1: Overview of the focused zero-shot problem for DocNMT. We study transferring contextual modeling from document-rich (teacher) languages to document-poor (student) languages, where only sentence pairs are given for students. The transfer occurs in a multilingual setup, shown by the dashed rectangles. Dashed arrows show the transfer direction. 2018). Most efforts on DocNMT aim at improv- ing contextual modeling via dedicated model ar- chitectures and/or decoding algorithms (Bawden et al., 2018; Voita et al., 2019; Chen et al., 2020) and heavily rely on large-scale parallel document resources. Nevertheless, document resources are unevenly distributed across language pairs, with most pairs having little to no such resources.1 One promising way to accommodate languages with varied training data is multilingual modeling, as demonstrated in multilingual SenNMT (Firat et al., 2016; Johnson et al., 2017). By sharing parameters across languages, multilingual model- ing encourages cross-lingual knowledge transfer, enabling performance improvement and even zero- shot transfer …

Content truncated to 500 words