Online Versus Offline NMT Quality: An In-depth Analysis on English-German and German-English

Proceedings of the 28th International Conference on Computational Linguistics, pages 5047–5058 Barcelona, Spain (Online), December 8-13, 2020 5047 Online Versus Offline NMT Quality: An In-depth Analysis on English–German and German–English Maha Elbayad1,2 Michael Ustaszewski3 Emmanuelle Esperança-Rodier1 Francis Brunet Manquat1 Jakob Verbeek4 Laurent Besacier1 1LIG - Université Grenoble Alpes, France 2Inria - Grenoble, France 3 University of Innsbruck, Department of Translation Studies 4 Facebook AI Research Abstract We conduct in this work an evaluation study comparing offline and online neural machine transla- tion architectures. Two sequence-to-sequence models: convolutional Pervasive Attention (Elbayad et al., 2018) and attention-based Transformer (Vaswani et al., 2017) are considered. We investigate, for both architectures, the impact of online decoding constraints on the translation quality through a carefully designed human evaluation on English-German and German-English language pairs, the latter being particularly sensitive to latency constraints. The evaluation results allow us to identify the strengths and shortcomings of each model when we shift to the online setup. 1 Introduction Sequence-to-Sequence models are state-of-the-art in a variety of sequence transduction tasks including machine translation (MT). The most widespread models are composed of an encoder that reads the entire source sequence, while a decoder (often equipped with an attention mechanism) iteratively produces the next target token given the full source and the decoded prefix. Aside from the conventional offline use case, recent works adapt sequence-to-sequence models for online (also referred to as simultaneous) decoding with low-latency constraints (Gu et al., 2017; Dalvi et al., 2018; Ma et al., 2019; Arivazhagan et al., 2019). Online decoding is desirable for applications such as real-time speech-to-speech interpretation. In such scenarios, the decoding process starts before the entire input sequence is available, and online prediction generally comes at the cost of reduced translation quality. In this work we focus on online neural machine translation (NMT) with deterministic wait-k decoding policies (Dalvi et al., 2018; Ma et al., 2019). With such a policy, we first read k tokens from the source then alternate between producing a target token and reading another source token (see Figure 1). We consider two sequence-to-sequence models, a position-based convolutional model and a content-based model with self-attention. We specifically use the recent convolutional Pervasive Attention (Elbayad et al., 2018) and Transformer (Vaswani et al., 2017). We investigate, for both architectures, the impact of online decoding constraints on the translation quality through a carefully designed human evaluation on English )German and German )English language pairs. Our contributions are twofold: (1) our work, to the best of our knowledge, is the first human evaluation of online vs. offline NMT systems. (2) We compare Transformer and Pervasive Attention architectures highlighting the advantages and shortcomings of each when we shift to the online setup. The rest of this paper is organized as follows: we present in §2 related work pertaining to online MT and error analysis of NMT systems. We describe our experimental setup for human evaluation and error analysis in §3. We follow with the evaluation results in §4 and summarize our findings in §5. …

Content truncated to 500 words