Continuous Space Translation Models with Neural Networks

2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 39–48, Montr´eal, Canada, June 3-8, 2012. c⃝2012 Association for Computational Linguistics Continuous Space Translation Models with Neural Networks Le Hai Son and Alexandre Allauzen and Franc¸ois Yvon Univ. Paris-Sud, France and LIMSI/CNRS rue John von Neumann, 91403 Orsay cedex, France Firstname.Lastname@limsi.fr Abstract The use of conventional maximum likelihood estimates hinders the performance of existing phrase-based translation models. For lack of sufficient training data, most models only con- sider a small amount of context. As a par- tial remedy, we explore here several contin- uous space translation models, where transla- tion probabilities are estimated using a con- tinuous representation of translation units in lieu of standard discrete representations. In order to handle a large set of translation units, these representations and the associated esti- mates are jointly computed using a multi-layer neural network with a SOUL architecture. In small scale and large scale English to French experiments, we show that the resulting mod- els can effectively be trained and used on top of a n-gram translation system, delivering sig- nificant improvements in performance. 1 Introduction The phrase-based approach to statistical machine translation (SMT) is based on the following infer- ence rule, which, given a source sentence s, selects the target sentence t and the underlying alignment a maximizing the following term: P(t, a|s) = 1 Z(s) exp � K � k=1 λkfk(s, t, a) � , (1) where K feature functions (fk) are weighted by a set of coefficients (λk), and Z is a normalizing fac- tor. The phrase-based approach differs from other approaches by the hidden variables of the translation process: the segmentation of a parallel sentence pair into phrase pairs and the associated phrase align- ments. This formulation was introduced in (Zens et al., 2002) as an extension of the word based mod- els (Brown et al., 1993), then later motivated within a discriminative framework (Och and Ney, 2004). One motivation for integrating more feature func- tions was to improve the estimation of the translation model P(t|s), which was initially based on relative frequencies, thus yielding poor estimates. This is because the units of phrase-based mod- els are phrase pairs, made of a source and a tar- get phrase; such units are viewed as the events of discrete random variables. The resulting representa- tions of phrases (or words) thus entirely ignore the morphological, syntactic and semantic relationships that exist among those units in both languages. This lack of structure hinders the generalization power of the model and reduces its ability to adapt to other domains. Another consequence is that phrase-based models usually consider a very restricted context1. This is a general issue in statistical Natural Lan- guage Processing (NLP) and many possible reme- dies have been proposed in the literature, such as, for instance, using smoothing techniques (Chen and Goodman, 1996), or working with linguistically en- riched, or more abstract, representations. In statisti- cal language modeling, another line of research con- siders …

Content truncated to 500 words