A Recursive Recurrent Neural Network for Statistical Machine Translation

Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, pages 1491–1500, Baltimore, Maryland, USA, June 23-25 2014. c⃝2014 Association for Computational Linguistics A Recursive Recurrent Neural Network for Statistical Machine Translation Shujie Liu1, Nan Yang2, Mu Li1 and Ming Zhou1 1Microsoft Research Asia, Beijing, China 2University of Science and Technology of China, Hefei, China shujliu, v-nayang, muli, mingzhou@microsoft.com Abstract In this paper, we propose a novel recursive recurrent neural network (R2NN) to mod- el the end-to-end decoding process for s- tatistical machine translation. R2NN is a combination of recursive neural network and recurrent neural network, and in turn integrates their respective capabilities: (1) new information can be used to generate the next hidden state, like recurrent neu- ral networks, so that language model and translation model can be integrated natu- rally; (2) a tree structure can be built, as recursive neural networks, so as to gener- ate the translation candidates in a bottom up manner. A semi-supervised training ap- proach is proposed to train the parameter- s, and the phrase pair embedding is ex- plored to model translation confidence di- rectly. Experiments on a Chinese to En- glish translation task show that our pro- posed R2NN can outperform the state- of-the-art baseline by about 1.5 points in BLEU. 1 Introduction Deep Neural Network (DNN), which essential- ly is a multi-layer neural network, has re-gained more and more attentions these years. With the efficient training methods, such as (Hinton et al., 2006), DNN is widely applied to speech and im- age processing, and has achieved breakthrough re- sults (Kavukcuoglu et al., 2010; Krizhevsky et al., 2012; Dahl et al., 2012). Applying DNN to natural language processing (NLP), representation or embedding of words is usually learnt first. Word embedding is a dense, low dimensional, real-valued vector. Each dimen- sion of the vector represents a latent aspect of the word, and captures its syntactic and semantic properties (Bengio et al., 2006). Word embedding is usually learnt from large amount of monolin- gual corpus at first, and then fine tuned for spe- cial distinct tasks. Collobert et al. (2011) propose a multi-task learning framework with DNN for various NLP tasks, including part-of-speech tag- ging, chunking, named entity recognition, and se- mantic role labelling. Recurrent neural networks are leveraged to learn language model, and they keep the history information circularly inside the network for arbitrarily long time (Mikolov et al., 2010). Recursive neural networks, which have the ability to generate a tree structured output, are ap- plied to natural language parsing (Socher et al., 2011), and they are extended to recursive neural tensor networks to explore the compositional as- pect of semantics (Socher et al., 2013). DNN is also introduced to Statistical Machine Translation (SMT) to learn several components or features of conventional framework, includ- ing word alignment, language modelling, transla- tion modelling and distortion modelling. Yang et al. (2013) adapt and extend the CD-DNN-HMM (Dahl et al., 2012) method to HMM-based word alignment model. In their work, bilingual word embedding is …

Content truncated to 500 words