Is Encoder-Decoder Redundant for Neural Machine Translation?

Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 562–574 November 20–23, 2022. ©2022 Association for Computational Linguistics 562 Is Encoder-Decoder Redundant for Neural Machine Translation? Yingbo Gao Christian Herold Zijian Yang Hermann Ney Human Language Technology and Pattern Recognition Group Computer Science Department RWTH Aachen University D-52056 Aachen, Germany {ygao|herold|zyang|ney}@cs.rwth-aachen.de Abstract Encoder-decoder architecture is widely adopted for sequence-to-sequence modeling tasks. For machine translation, despite the evolution from long short-term memory networks to Trans- former networks, plus the introduction and de- velopment of attention mechanism, encoder- decoder is still the de facto neural network ar- chitecture for state-of-the-art models. While the motivation for decoding information from some hidden space is straightforward, the strict separation of the encoding and decoding steps into an encoder and a decoder in the model ar- chitecture is not necessarily a must. Compared to the task of autoregressive language model- ing in the target language, machine translation simply has an additional source sentence as con- text. Given the fact that neural language mod- els nowadays can already handle rather long contexts in the target language, it is natural to ask whether simply concatenating the source and target sentences and training a language model to do translation would work. In this work, we investigate the aforementioned con- cept for machine translation. Specifically, we experiment with bilingual translation, transla- tion with additional target monolingual data, and multilingual translation. In all cases, this alternative approach performs on par with the baseline encoder-decoder Transformer, suggest- ing that an encoder-decoder architecture might be redundant for neural machine translation. 1 Introduction Sequence-to-sequence modeling is often ap- proached with Neural Networks (NNs), promi- nently encoder-decoder NNs, nowadays. For the task of Machine Translation (MT), which is by definition also a sequence-to-sequence task, the default choice of NN topology is also an encoder- decoder architecture. For example, in early works like Kalchbrenner and Blunsom (2013), the authors already make the distinction between their convo- lutional sentence model (encoder) and recurrent language model (decoder) conditioned on the for- mer. In follow-up works like Sutskever et al. (2014) and Cho et al. (2014a,b), the concept of encoder- decoder network is further developed. While ex- tensions such as attention (Bahdanau et al., 2014), multi-task learning (Luong et al., 2015), convo- lutional networks (Gehring et al., 2017) and self- attention (Vaswani et al., 2017) are considered for sequence-to-sequence learning, the idea of encod- ing information into some hidden space and decod- ing from that hidden representation sticks around. Given the success and wide popularity of the Transformer network (Vaswani et al., 2017), many works focus on understanding and improving indi- vidual components, e.g. positional encoding (Shaw et al., 2018), multi-head attention (Voita et al., 2019), and an alignment interpretation of cross attention (Alkhouli et al., 2018). In works that go a bit further and make bigger changes in terms of modeling, e.g. performing round-trip translation (Tu et …

Content truncated to 500 words