Recurrent Continuous Translation Models

Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1700–1709, Seattle, Washington, USA, 18-21 October 2013. c⃝2013 Association for Computational Linguistics Recurrent Continuous Translation Models Nal Kalchbrenner Phil Blunsom Department of Computer Science University of Oxford {nal.kalchbrenner,phil.blunsom}@cs.ox.ac.uk Abstract We introduce a class of probabilistic con- tinuous translation models called Recur- rent Continuous Translation Models that are purely based on continuous representations for words, phrases and sentences and do not rely on alignments or phrasal translation units. The models have a generation and a condi- tioning aspect. The generation of the transla- tion is modelled with a target Recurrent Lan- guage Model, whereas the conditioning on the source sentence is modelled with a Convolu- tional Sentence Model. Through various ex- periments, we show first that our models ob- tain a perplexity with respect to gold transla- tions that is > 43% lower than that of state- of-the-art alignment-based translation models. Secondly, we show that they are remarkably sensitive to the word order, syntax, and mean- ing of the source sentence despite lacking alignments. Finally we show that they match a state-of-the-art system when rescoring n-best lists of translations. 1 Introduction In most statistical approaches to machine transla- tion the basic units of translation are phrases that are composed of one or more words. A crucial com- ponent of translation systems are models that esti- mate translation probabilities for pairs of phrases, one phrase being from the source language and the other from the target language. Such models count phrase pairs and their occurrences as distinct if the surface forms of the phrases are distinct. Although distinct phrase pairs often share significant similari- ties, linguistic or otherwise, they do not share statis- tical weight in the models’ estimation of their trans- lation probabilities. Besides ignoring the similar- ity of phrase pairs, this leads to general sparsity is- sues. The estimation is sparse or skewed for the large number of rare or unseen phrase pairs, which grows exponentially in the length of the phrases, and the generalisation to other domains is often limited. Continuous representations have shown promise at tackling these issues. Continuous representations for words are able to capture their morphological, syntactic and semantic similarity (Collobert and We- ston, 2008). They have been applied in continu- ous language models demonstrating the ability to overcome sparsity issues and to achieve state-of-the- art performance (Bengio et al., 2003; Mikolov et al., 2010). Word representations have also shown a marked sensitivity to conditioning information (Mikolov and Zweig, 2012). Continuous repre- sentations for characters have been deployed in character-level language models demonstrating no- table language generation capabilities (Sutskever et al., 2011). Continuous representations have also been constructed for phrases and sentences. The rep- resentations are able to carry similarity and task de- pendent information, e.g. sentiment, paraphrase or dialogue labels, significantly beyond the word level and to accurately predict labels for a highly diverse range of unseen phrases and sentences (Grefenstette et al., 2011; Socher et al., 2011; Socher et al., 2012; …

Content truncated to 500 words