Diverse Pretrained Context Encodings Improve Document Translation

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 1299–1311 August 1–6, 2021. ©2021 Association for Computational Linguistics 1299 Diverse Pretrained Context Encodings Improve Document Translation Domenic Donato, Lei Yu, Chris Dyer DeepMind London, United Kingdom {domenicd,leiyu,cdyer}@deepmind.com Abstract We propose a new architecture for adapting a sentence-level sequence-to-sequence trans- former by incorporating multiple pretrained document context signals and assess the im- pact on translation performance of (1) dif- ferent pretraining approaches for generating these signals, (2) the quantity of parallel data for which document context is available, and (3) conditioning on source, target, or source and target contexts. Experiments on the NIST Chinese–English, and IWSLT and WMT English–German tasks support four gen- eral conclusions: that using pretrained con- text representations markedly improves sam- ple efficiency, that adequate parallel data re- sources are crucial for learning to use doc- ument context, that jointly conditioning on multiple context representations outperforms any single representation, and that source con- text is more valuable for translation perfor- mance than target side context. Our best multi- context model consistently outperforms the best existing context-aware transformers. 1 Introduction Generating an adequate translation for a sen- tence often requires understanding the context in which the sentence occurs (and in which its translation will occur). Although single-sentence translation models demonstrate remarkable perfor- mance (Chen et al., 2018; Vaswani et al., 2017; Bahdanau et al., 2015), extra-sentential informa- tion can be necessary to make correct decisions about lexical choice, tense, pronominal usage, and stylistic features, and therefore designing models capable of using this information is a necessary step towards fully automatic high-quality translation. A series of papers have developed architectures that permit the broader translation model to condition on extra-sentential context (Zhang et al., 2018; Mi- culicich et al., 2018), operating jointly on multiple sentences at once (Junczys-Dowmunt, 2019), or indirectly conditioning on target side document context using Bayes’ rule (Yu et al., 2020b). While noteworthy progress has been made at modeling monolingual documents (Brown et al., 2020), progress on document translation has been less remarkable, and continues to be hampered by the limited quantities of parallel document data relative to the massive quantities of monolingual document data. One recurring strategy for deal- ing with this data scarcity—and which is the ba- sis for this work—is to adapt a sentence-level sequence-to-sequence model by making additional document context available in a second stage of training (Maruf et al., 2019; Zhang et al., 2018; Miculicich et al., 2018; Haffari and Maruf, 2018). This two-stage training approach provides an in- ductive bias that encourages the learner to explain translation decisions preferentially in terms of the current sentence being translated, but these can be modulated at the margins by using document con- text. However, a weakness of this approach is that the conditional dependence of a translation on its surrounding context given the source sentence is weak, and learning good context representations purely on the basis of scarce parallel document …

Content truncated to 500 words