Minimum Translation Modeling with Recurrent Neural Networks

Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 20–29, Gothenburg, Sweden, April 26-30 2014. c⃝2014 Association for Computational Linguistics Minimum Translation Modeling with Recurrent Neural Networks Yuening Hu Department of Computer Science University of Maryland, College Park ynhu@cs.umd.edu Michael Auli, Qin Gao, Jianfeng Gao Microsoft Research Redmond, WA, USA {michael.auli,qigao,jfgao}@microsoft.com Abstract We introduce recurrent neural network- based Minimum Translation Unit (MTU) models which make predictions based on an unbounded history of previous bilin- gual contexts. Traditional back-off n-gram models suffer under the sparse nature of MTUs which makes estimation of high- order sequence models challenging. We tackle the sparsity problem by modeling MTUs both as bags-of-words and as a sequence of individual source and target words. Our best results improve the out- put of a phrase-based statistical machine translation system trained on WMT 2012 French-English data by up to 1.5 BLEU, and we outperform the traditional n-gram based MTU approach by up to 0.8 BLEU. 1 Introduction Classical phrase-based translation models rely heavily on the language model and the re- ordering model to capture dependencies between phrases. Sequence models over Minimum Trans- lation Units (MTUs) have been shown to com- plement both syntax-based (Quirk and Menezes, 2006) as well as phrase-based (Zhang et al., 2013) models by explicitly modeling relationships be- tween phrases. MTU models have been tradi- tionally estimated using standard back-off n-gram techniques (Quirk and Menezes, 2006; Crego and Yvon, 2010; Zhang et al., 2013), similar to word- based language models (§2). However, the estimation of higher-order n-gram models becomes increasingly difficult due to data sparsity issues associated with large n-grams, even when training on over one hundred billion words (Heafield et al., 2013); bilingual units are much sparser than words and are therefore even harder to estimate. Another drawback of n-gram mod- els is that future predictions are based on a limited amount of previous context that is often not suf- ficient to capture important aspects of human lan- guage (Rastrow et al., 2012). Recently, several feed-forward neural network- based models have achieved impressive improve- ments over traditional back-off n-gram models in language modeling (Bengio et al., 2003; Schwenk et al., 2007; Schwenk et al., 2012; Vaswani et al., 2013), as well as translation modeling (Allauzen et al., 2011; Le et al., 2012; Gao et al., 2013). These models tackle the data sparsity problem by rep- resenting words in continuous space rather than as discrete units. Similar words are grouped in the same sub-space rather than being treated as separate entities. Neural network models can be seen as functions over continuous representations exploiting the similarity between words, thereby making the estimation of probabilities over higher- order n-grams easier. However, feed-forward networks do not directly address the limited context issue either, since pre- dictions are based on a fixed-size context, similar to back-off n-gram models. We therefore focus in this paper on recurrent neural network architec- tures, which address the limited context issue by basing predictions on an unbounded history …

Content truncated to 500 words