Joint Language and Translation Modeling with Recurrent Neural Networks

Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1044–1054, Seattle, Washington, USA, 18-21 October 2013. c⃝2013 Association for Computational Linguistics Joint Language and Translation Modeling with Recurrent Neural Networks Michael Auli, Michel Galley, Chris Quirk, Geoffrey Zweig Microsoft Research Redmond, WA, USA {michael.auli,mgalley,chrisq,gzweig}@microsoft.com Abstract We present a joint language and transla- tion model based on a recurrent neural net- work which predicts target words based on an unbounded history of both source and tar- get words. The weaker independence as- sumptions of this model result in a vastly larger search space compared to related feed- forward-based language or translation models. We tackle this issue with a new lattice rescor- ing algorithm and demonstrate its effective- ness empirically. Our joint model builds on a well known recurrent neural network language model (Mikolov, 2012) augmented by a layer of additional inputs from the source language. We show competitive accuracy compared to the traditional channel model features. Our best results improve the output of a system trained on WMT 2012 French-English data by up to 1.5 BLEU, and by 1.1 BLEU on average across several test sets. 1 Introduction Recently, several feed-forward neural network- based language and translation models have achieved impressive accuracy improvements on sta- tistical machine translation tasks (Allauzen et al., 2011; Le et al., 2012b; Schwenk et al., 2012). In this paper we focus on recurrent neural network archi- tectures, which have recently advanced the state of the art in language modeling (Mikolov et al., 2010; Mikolov et al., 2011a; Mikolov, 2012), outperform- ing multi-layer feed-forward based networks in both perplexity and word error rate in speech recognition (Arisoy et al., 2012; Sundermeyer et al., 2013). The major attraction of recurrent architectures is their potential to capture long-span dependencies since predictions are based on an unbounded history of previous words. This is in contrast to feed-forward networks as well as conventional n-gram models, both of which are limited to fixed-length contexts. Building on the success of recurrent architectures, we base our joint language and translation model on an extension of the recurrent neural network lan- guage model (Mikolov and Zweig, 2012) that intro- duces a layer of additional inputs (§2). Most previous work on neural networks for speech recognition or machine translation used a rescoring setup based on n-best lists (Arisoy et al., 2012; Mikolov, 2012) for evaluation, thereby side stepping the algorithmic and engineering challenges of direct decoder-integration.1 Instead, we exploit lattices, which offer a much richer representation of the decoder output, since they compactly encode an exponential number of translation hypotheses in polynomial space. In contrast, n-best lists are typi- cally very redundant, representing only a few com- binations of top scoring arcs in the lattice. A major challenge in lattice rescoring with a recurrent neural network model is the effect of the unbounded history on search since the usual dynamic programming as- sumptions which are exploited for efficiency do not hold up anymore. We apply a novel algorithm to the task of …

Content truncated to 500 words