Translation Modeling with Bidirectional Recurrent Neural Networks

Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 14–25, October 25-29, 2014, Doha, Qatar. c⃝2014 Association for Computational Linguistics Translation Modeling with Bidirectional Recurrent Neural Networks Martin Sundermeyer1, Tamer Alkhouli1, Joern Wuebker1, and Hermann Ney1,2 1Human Language Technology and Pattern Recognition Group RWTH Aachen University, Aachen, Germany 2Spoken Language Processing Group Univ. Paris-Sud, France and LIMSI/CNRS, Orsay, France {surname}@cs.rwth-aachen.de Abstract This work presents two different trans- lation models using recurrent neural net- works. The first one is a word-based ap- proach using word alignments. Second, we present phrase-based translation mod- els that are more consistent with phrase- based decoding. Moreover, we introduce bidirectional recurrent neural models to the problem of machine translation, allow- ing us to use the full source sentence in our models, which is also of theoretical inter- est. We demonstrate that our translation models are capable of improving strong baselines already including recurrent neu- ral language models on three tasks: IWSLT 2013 German→English, BOLT Arabic→English and Chinese→English. We obtain gains up to 1.6% BLEU and 1.7% TER by rescoring 1000-best lists. 1 Introduction Neural network models have recently experienced unprecedented attention in research on statistical machine translation (SMT). Several groups have reported strong improvements over state-of-the-art baselines using feedforward neural network-based language models (Schwenk et al., 2006; Vaswani et al., 2013), as well as translation models (Le et al., 2012; Schwenk, 2012; Devlin et al., 2014). Different from the feedforward design, recurrent neural networks (RNNs) have the advantage of be- ing able to take into account an unbounded his- tory of previous observations. In theory, this en- ables them to model long-distance dependencies of arbitrary length. However, while previous work on translation modeling with recurrent neural net- works shows its effectiveness on standard base- lines, so far no notable gains have been presented on top of recurrent language models (Auli et al., 2013; Kalchbrenner and Blunsom, 2013; Hu et al., 2014). In this work, we present two novel approaches to recurrent neural translation modeling: word- based and phrase-based. The word-based ap- proach assumes one-to-one aligned source and target sentences. We evaluate different ways of resolving alignment ambiguities to obtain such alignments. The phrase-based RNN approach is more closely tied to the underlying translation paradigm. It models actual phrasal translation probabilities while avoiding sparsity issues by us- ing single words as input and output units. Fur- thermore, in addition to the unidirectional formu- lation, we are the first to propose a bidirectional architecture which can take the full source sen- tence into account for all predictions. Our ex- periments show that these models can improve state-of-the-art baselines containing a recurrent language model on three tasks. For our compet- itive IWSLT 2013 German→English system, we observe gains of up to 1.6% BLEU and 1.7% TER. Improvements are also demonstrated on top of our evaluation systems for BOLT Arabic→English and Chinese→English, which also include recur- rent neural language models. The rest of this paper is structured as follows. In Section 2 we review related …

Content truncated to 500 words