A Convolutional Encoder Model for Neural Machine Translation

Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 123–135 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1012 Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 123–135 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1012 A Convolutional Encoder Model for Neural Machine Translation Jonas Gehring, Michael Auli, David Grangier, Yann N. Dauphin Facebook AI Research Abstract The prevalent approach to neural machine translation relies on bi-directional LSTMs to encode the source sentence. We present a faster and simpler architecture based on a succession of convolutional layers. This al- lows to encode the source sentence simulta- neously compared to recurrent networks for which computation is constrained by tem- poral dependencies. On WMT’16 English- Romanian translation we achieve compet- itive accuracy to the state-of-the-art and on WMT’15 English-German we outper- form several recently published results. Our models obtain almost the same accuracy as a very deep LSTM setup on WMT’14 English-French translation. We speed up CPU decoding by more than two times at the same or higher accuracy as a strong bi- directional LSTM.1 1 Introduction Neural machine translation (NMT) is an end-to-end approach to machine translation (Sutskever et al., 2014). The most successful approach to date en- codes the source sentence with a bi-directional re- current neural network (RNN) into a variable length representation and then generates the translation left-to-right with another RNN where both com- ponents interface via a soft-attention mechanism (Bahdanau et al., 2015; Luong et al., 2015a; Brad- bury and Socher, 2016; Sennrich et al., 2016a). Recurrent networks are typically parameterized as long short term memory networks (LSTM; Hochre- iter et al. 1997) or gated recurrent units (GRU; Cho et al. 2014), often with residual or skip connec- tions (Wu et al., 2016; Zhou et al., 2016) to enable stacking of several layers (§2). There have been several attempts to use convo- lutional encoder models for neural machine trans- 1The source code will be availabe at https://github. com/facebookresearch/fairseq lation in the past but they were either only ap- plied to rescoring n-best lists of classical systems (Kalchbrenner and Blunsom, 2013) or were not competitive to recurrent alternatives (Cho et al., 2014a). This is despite several attractive properties of convolutional networks. For example, convolu- tional networks operate over a fixed-size window of the input sequence which enables the simultaneous computation of all features for a source sentence. This contrasts to RNNs which maintain a hidden state of the entire past that prevents parallel com- putation within a sequence. A succession of convolutional layers provides a shorter path to capture relationships between ele- ments of a sequence compared to RNNs.2 This also eases learning because the resulting tree-structure applies a fixed number of non-linearities compared to a recurrent neural network for which the number of non-linearities vary depending on the time-step. Because processing is bottom-up, all words un- dergo the same number of transformations, whereas for RNNs the first …

Content truncated to 500 words