Investigations on Phrase-based Decoding with Recurrent Neural Network Language and Translation Models

Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 294–303, Lisboa, Portugal, 17-18 September 2015. c⃝2015 Association for Computational Linguistics. Investigations on Phrase-based Decoding with Recurrent Neural Network Language and Translation Models Tamer Alkhouli, Felix Rietig, and Hermann Ney Human Language Technology and Pattern Recognition Group RWTH Aachen University, Aachen, Germany {surname}@cs.rwth-aachen.de Abstract This work explores the application of re- current neural network (RNN) language and translation models during phrase- based decoding. Due to their use of un- bounded context, the decoder integration of RNNs is more challenging compared to the integration of feedforward neural mod- els. In this paper, we apply approxima- tions and use caching to enable RNN de- coder integration, while requiring reason- able memory and time resources. We an- alyze the effect of caching on translation quality and speed, and use it to integrate RNN language and translation models into a phrase-based decoder. To the best of our knowledge, no previous work has dis- cussed the integration of RNN translation models into phrase-based decoding. We also show that a special RNN can be inte- grated efficiently without the need for ap- proximations. We compare decoding us- ing RNNs to rescoring n-best lists on two tasks: IWSLT 2013 German→English, and BOLT Arabic→English. We demon- strate that the performance of decoding with RNNs is at least as good as using them in rescoring. 1 Introduction Applying neural networks to statistical machine translation has been gaining increasing attention recently. Neural network language and translation models have been successfully applied to rescore the first-pass decoding output (Le et al., 2012; Sundermeyer et al., 2014; Hu et al., 2014; Guta et al., 2015). These models include feedforward and recurrent neural networks. A more ambitious move is to apply neural net- works directly during decoding, which in principle should give the models a better chance to influence translation in comparison to rescoring, as rescor- ing is limited to scoring and reranking fixed n- best lists. Recently, neural networks were used for standalone decoding using a simple beam-search word-based decoder (Sutskever et al., 2014; Bah- danau et al., 2015). Another approach is to apply neural models directly in a phrase-based decoder. We focus on this approach, which is challeng- ing since phrase-based decoding typically involves generating tens or even hundreds of millions of partial hypotheses. Scoring such a number of hy- potheses using neural models is expensive, mainly due to the usually large output layer. Nevertheless, decoder integration has been done in (Vaswani et al., 2013) for feedforward neural language models. Devlin et al. (2014) integrate feedforward transla- tion models into phrase-based decoding reporting major improvements, which highlight the strength of the underlying models. In related fields like e. g. language model- ing, RNNs has been shown to perform consid- erably better than standard feedforward architec- tures (Mikolov et al., 2011; Arisoy et al., 2012; Sundermeyer et al., 2013; Liu et al., 2014). Sun- dermeyer et al. (2014) also show that RNN trans- lation models outperform feedforward networks in rescoring. Given the …

Content truncated to 500 words