Automatic Discrimination of Human and Neural Machine Translation: A Study with Multiple Pre-Trained Models and Longer Context

Rethinking the Design of Sequence-to-Sequence Models for Efficient Machine Translation Maha Elbayad† LIG - Universit´e Grenoble Alpes, France Inria - Grenoble, France maha.elbayad@inria.fr In recent years, deep learning has enabled im- pressive achievements in Machine Translation. Neural Machine Translation (NMT) relies on train- ing deep neural networks with large number of pa- rameters on vast amounts of parallel data to learn how to translate from one language to another. One crucial factor to the success of NMT is the de- sign of new powerful and efficient architectures. State-of-the-art systems are encoder-decoder mod- els (Cho et al., 2014; Sutskever et al., 2014; Bah- danau et al., 2015; Gehring et al., 2017; Vaswani et al., 2017) that first encode a source sequence into a set of feature vectors and then decode the target se- quence conditioning on the source features. In this thesis we question the encoder-decoder paradigm and advocate for an intertwined encoding of the source and target so that the two sequences inter- act at increasing levels of abstraction. For this pur- pose, we introduce Pervasive Attention, an NMT model with a computational graph different from existing encoder-decoder models. In Pervasive at- tention, the source and the target communicate and interact throughout the encoding process towards abstract features. To this end, our NMT model uses two-dimensional convolutional neural networks to process a grid of features where every position rep- resents an interaction between a target and a source tokens. To tackle a different aspect of efficiency in NMT systems, we explore the challenging task of on- line (also called simultaneous) machine transla- tion (F¨ugen et al., 2007; Mieno et al., 2015; Dalvi et al., 2018; Ma et al., 2019) where the source is † Now at Meta AI Research c⃝ 2022 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC- BY-ND. read incrementally and the decoder is fed partial contexts so that the model can alternate between reading and writing. To improve the translation’s delay in online NMT systems, we first setup a com- mon framework for online sequence-to-sequence models that will allow us to train existing de- terministic decoders that alternate between read- ing the source and writing the target in a pre- determined fashion, and dynamic decoders that condition their decoding path on the current input. We first prove the effectiveness of the determin- istic online decoders and their ability to perform well outside the delay range they were optimized for. We then adapt Pervasive Attention models for the task of online translation with both a determin- istic and a dynamic decoding strategy. We also address the resource-efficiency of encoder-decoder models, namely Transformer models (Vaswani et al., 2017), state-of-the-art in a wide range of NLP tasks (Devlin et al., 2019; Yang et al., 2019; Liu et al., 2019; Ng et al., 2019). Models based on the Transformer architecture can grow deep, accumulating billions of parameters. We posit that going deeper in a neural network is not required for all instances, …

Content truncated to 500 words