Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation

Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1724–1734, October 25-29, 2014, Doha, Qatar. c⃝2014 Association for Computational Linguistics Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation Kyunghyun Cho Bart van Merri¨enboer Caglar Gulcehre Universit´e de Montr´eal firstname.lastname@umontreal.ca Dzmitry Bahdanau Jacobs University, Germany d.bahdanau@jacobs-university.de Fethi Bougares Holger Schwenk Universit´e du Maine, France firstname.lastname@lium.univ-lemans.fr Yoshua Bengio Universit´e de Montr´eal, CIFAR Senior Fellow find.me@on.the.web Abstract In this paper, we propose a novel neu- ral network model called RNN Encoder– Decoder that consists of two recurrent neural networks (RNN). One RNN en- codes a sequence of symbols into a fixed- length vector representation, and the other decodes the representation into another se- quence of symbols. The encoder and de- coder of the proposed model are jointly trained to maximize the conditional prob- ability of a target sequence given a source sequence. The performance of a statisti- cal machine translation system is empiri- cally found to improve by using the con- ditional probabilities of phrase pairs com- puted by the RNN Encoder–Decoder as an additional feature in the existing log-linear model. Qualitatively, we show that the proposed model learns a semantically and syntactically meaningful representation of linguistic phrases. 1 Introduction Deep neural networks have shown great success in various applications such as objection recognition (see, e.g., (Krizhevsky et al., 2012)) and speech recognition (see, e.g., (Dahl et al., 2012)). Fur- thermore, many recent works showed that neu- ral networks can be successfully used in a num- ber of tasks in natural language processing (NLP). These include, but are not limited to, language modeling (Bengio et al., 2003), paraphrase detec- tion (Socher et al., 2011) and word embedding ex- traction (Mikolov et al., 2013). In the field of sta- tistical machine translation (SMT), deep neural networks have begun to show promising results. (Schwenk, 2012) summarizes a successful usage of feedforward neural networks in the framework of phrase-based SMT system. Along this line of research on using neural net- works for SMT, this paper focuses on a novel neu- ral network architecture that can be used as a part of the conventional phrase-based SMT system. The proposed neural network architecture, which we will refer to as an RNN Encoder–Decoder, con- sists of two recurrent neural networks (RNN) that act as an encoder and a decoder pair. The en- coder maps a variable-length source sequence to a fixed-length vector, and the decoder maps the vec- tor representation back to a variable-length target sequence. The two networks are trained jointly to maximize the conditional probability of the target sequence given a source sequence. Additionally, we propose to use a rather sophisticated hidden unit in order to improve both the memory capacity and the ease of training. The proposed RNN Encoder–Decoder with a novel hidden unit is empirically evaluated on the task of translating from English to French. We train the model to learn the translation probabil- ity of an English phrase to a corresponding French phrase. The model is then …

Content truncated to 500 words