LSTM Neural Reordering Feature for Statistical Machine Translation

Proceedings of NAACL-HLT 2016, pages 977–982, San Diego, California, June 12-17, 2016. c⃝2016 Association for Computational Linguistics LSTM Neural Reordering Feature for Statistical Machine Translation Yiming Cui, Shijin Wang and Jianfeng Li iFLYTEK Research, Beijing, China {ymcui,sjwang3,jfli3}@iflytek.com Abstract Artificial neural networks are powerful mod- els, which have been widely applied into many aspects of machine translation, such as language modeling and translation mod- eling. Though notable improvements have been made in these areas, the reordering prob- lem still remains a challenge in statistical ma- chine translations. In this paper, we present a novel neural reordering model that directly models word pairs and their alignment. Fur- ther by utilizing LSTM recurrent neural net- works, much longer context could be learned for reordering prediction. Experimental re- sults on NIST OpenMT12 Arabic-English and Chinese-English 1000-best rescoring task show that our LSTM neural reordering feature is robust, and achieves significant improve- ments over various baseline systems. 1 Introduction In statistical machine translation, the language model, translation model, and reordering model are the three most important components. Among these models, the reordering model plays an important role in phrase-based machine translation (Koehn et al., 2004), and it still remains a major challenge in current study. In recent years, various phrase reordering meth- ods have been proposed for phrase-based SMT sys- tems, which can be classified into two broad cate- gories: (1) Distance-based RM: Penalize phrase displace- ments with respect to the degree of non- monotonicity (Koehn et al., 2004). (2) Lexicalized RM: Conditions reordering proba- bilities on current phrase pairs. According to the orientation determinants, lexicalized reordering model can further be classified into word-based RM (Tillman, 2004), phrase-based RM (Koehn et al., 2007), and hierarchical phrase-based RM (Galley and Manning, 2008). Furthermore, some researchers proposed a re- ordering model that conditions both current and previous phrase pairs by utilizing recursive auto- encoders (Li et al., 2014). In this paper, we propose a novel neural reorder- ing feature by including longer context for pre- dicting orientations. We utilize a long short-term memory recurrent neural network (LSTM-RNN) (Graves, 1997), and directly models word pairs to predict its most probable orientation. Experimen- tal results on NIST OpenMT12 Arabic-English and Chinese-English translation show that our neural re- ordering model achieves significant improvements over various baselines in 1000-best rescoring task. 2 Related Work Recently, various neural network models have been applied into machine translation. Feed-forward neural language model was first proposed by Bengio et al. (2003), which was a breakthrough in language modeling. Mikolov et al. (2011) proposed to use recurrent neural net- work in language modeling, which can include much longer context history for predicting next word. Experimental results show that RNN-based language model significantly outperform standard feed-forward language model. 977 Devlin et al. (2014) proposed a neural network joint model (NNJM) by conditioning both source and target language context for target word predict- ing. Though the network architecture is a simple feed-forward neural network, the results have shown significant improvements over state-of-the-art base- lines. Sundermeyer et al. (2014) …

Content truncated to 500 words