A Comparison of Transformer and Recurrent Neural Networks on Multilingual Neural Machine Translation

Proceedings of the 27th International Conference on Computational Linguistics, pages 641–652 Santa Fe, New Mexico, USA, August 20-26, 2018. 641 A Comparison of Transformer and Recurrent Neural Networks on Multilingual Neural Machine Translation Surafel M. Lakew University of Trento Fondazione Bruno Kessler lakew@fbk.eu Mauro Cettolo Fondazione Bruno Kessler cettolo@fbk.eu Marcello Federico MMT Srl, Trento Fondazione Bruno Kessler federico@fbk.eu Abstract Recently, neural machine translation (NMT) has been extended to multilinguality, that is to han- dle more than one translation direction with a single system. Multilingual NMT showed compet- itive performance against pure bilingual systems. Notably, in low-resource settings, it proved to work effectively and efficiently, thanks to shared representation space that is forced across lan- guages and induces a sort of transfer-learning. Furthermore, multilingual NMT enables so-called zero-shot inference across language pairs never seen at training time. Despite the increasing in- terest in this framework, an in-depth analysis of what a multilingual NMT model is capable of and what it is not is still missing. Motivated by this, our work (i) provides a quantitative and com- parative analysis of the translations produced by bilingual, multilingual and zero-shot systems; (ii) investigates the translation quality of two of the currently dominant neural architectures in MT, which are the Recurrent and the Transformer ones; and (iii) quantitatively explores how the closeness between languages influences the zero-shot translation. Our analysis leverages mul- tiple professional post-edits of automatic translations by several different systems and focuses both on automatic standard metrics (BLEU and TER) and on widely used error categories, which are lexical, morphology, and word order errors. 1 Introduction As witnessed by recent machine translation evaluation campaigns (IWSLT (Cettolo et al., 2017), WMT (Bojar et al., 2017)), in the past few years several model variants and training procedures have been proposed and tested in neural machine translation (NMT). NMT models were mostly employed in conventional single language-pair settings, where the training process exploits a parallel corpus from a source language to a target language, and the inference involves only those two languages in the same direction. However, there have also been attempts to incorporate multiple languages in the source (Lu- ong et al., 2015a; Zoph and Knight, 2016; Lee et al., 2016), in the target (Dong et al., 2015), or in both sides like Firat et al. (2016) which combines a shared attention mechanism and multiple encoder-decoder layers. Regardless, the simple approach proposed in Johnson et al. (2016) and Ha et al. (2016) remains outstandingly effective: it relies on single “universal” encoder, decoder and attention modules, and man- ages multilinguality by introducing an artificial token at the beginning of the input sentence to specify the requested target language. The current NMT state-of-the-art includes the use of recurrent neural networks, initially introduced in Sutskever et al. (2014; Cho et al. (2014), convolutional neural networks, proposed by Gehring et al. (2017), and so-called transformer neural networks, recently proposed by Vaswani et al. (2017). All of them implement an encoder-decoder architecture, suitable for sequence-to-sequence tasks like machine translation, and …

Content truncated to 500 words