Deep Neural Machine Translation with Weakly-Recurrent Units

Deep Neural Machine Translation with Weakly-Recurrent Units Mattia A. Di Gangi FBK, Trento, Italy University of Trento, Italy digangi@fbk.eu Marcello Federico MMT Srl, Trento, Italy FBK, Trento, Italy federico@fbk.eu Abstract Recurrent neural networks (RNNs) have represented for years the state of the art in neural machine translation. Re- cently, new architectures have been pro- posed, which can leverage parallel compu- tation on GPUs better than classical RNNs. Faster training and inference combined with different sequence-to-sequence mod- eling also lead to performance improve- ments. While the new models completely depart from the original recurrent archi- tecture, we decided to investigate how to make RNNs more efficient. In this work, we propose a new recurrent NMT architec- ture, called Simple Recurrent NMT, built on a class of fast and weakly-recurrent units that use layer normalization and mul- tiple attentions. Our experiments on the WMT14 English-to-German and WMT16 English-Romanian benchmarks show that our model represents a valid alternative to LSTMs, as it can achieve better results at a significantly lower computational cost. 1 Introduction Neural machine translation (NMT) (Sutskever et al., 2014; Bahdanau et al., 2015) is a sequence-to- sequence problem that requires generating a sen- tence in a target language from a corresponding sentence in a source language. Similarly to other language processing task, NMT has mostly em- ployed recurrent neural networks (RNNs) (Sen- nrich et al., 2016b; Sennrich et al., 2017b; Luong c⃝ 2018 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC- BY-ND. and Manning, 2015), in both their LSTM (Hochre- iter and Schmidhuber, 1997) and GRU (Cho et al., 2014) variants, which can model long-range de- pendencies. Besides their simplicity, the choice of RNNs is also due to their expressive power, which has been proven equivalent to Turing Ma- chines (Siegelmann and Sontag, 1995). RNNs have represented so far the state of the art of ma- chine translation, and have constantly been en- hanced to improve their performance. Nonethe- less, their explicit time dependencies make train- ing of deep RNNs computationally very expensive (Wu et al., 2016; Barone et al., 2017). Recent works have proposed new NMT archi- tectures, not based on RNNs, that obtained sig- nificant improvements both in training speed and translation quality: the so-called convolutional sequence-to-sequence (Gehring et al., 2017) and the self-attentive or transformer (Vaswani et al., 2017) models. Speed improvements by these mod- els mainly come from the possibility of paralleliz- ing computations over word sequences, as both models do not have time dependencies. On the other hand, performance improvements appear to be due to the path lengths needed by the networks to connect distant words in a sentence: linear for RNNs, logarithmic for convolutional models, and constant for the transformer. In this paper we propose a neural architec- ture that shares some properties with the above- mentioned models, while maintaining a recurrent design. Our hypothesis is that current RNNs for NMT have not been designed to take full advan- tage of deep structures and that …

Content truncated to 500 words