Probabilistic Inference for Machine Translation

Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 215–223, Honolulu, October 2008. c⃝2008 Association for Computational Linguistics Probabilistic Inference for Machine Translation Phil Blunsom and Miles Osborne School of Informatics, University of Edinburgh 10 Crichton Street, Edinburgh, EH8 9AB, UK {pblunsom,miles}@inf.ed.ac.uk Abstract We advance the state-of-the-art for discrimi- natively trained machine translation systems by presenting novel probabilistic inference and search methods for synchronous gram- mars. By approximating the intractable space of all candidate translations produced by inter- secting an ngram language model with a synchronous grammar, we are able to train and decode models incorporating millions of sparse, heterogeneous features. Further, we demonstrate the power of the discriminative training paradigm by extracting structured syntactic features, and achieving increases in translation performance. 1 Introduction The goal of creating statistical machine translation (SMT) systems incorporating rich, sparse, features over syntax and morphology has consumed much recent research attention. Discriminative approaches are widely seen as a promising technique, poten- tially allowing us to further the state-of-the-art. Most work on discriminative training for SMT has focussed on linear models, often with margin based algorithms (Liang et al., 2006; Watanabe et al., 2006), or rescaling a product of sub-models (Och, 2003; Ittycheriah and Roukos, 2007). Recent work by Blunsom et al. (2008) has shown how translation can be framed as a probabilistic log-linear model, where the distribution over trans- lations is modelled in terms of a latent variable on derivations. Their approach was globally opti- mised and discriminative trained. However, a lan- guage model, an information source known to be crucial for obtaining good performance in SMT, was notably omitted. This was because adding a lan- guage model would mean that the normalising parti- tion function could no longer be exactly calculated, thereby preventing efficient parameter estimation. Here, we show how language models can be incorporated into large-scale discriminative transla- tion models, without losing the probabilistic inter- pretation of the model. The key insight is that we can use Monte-Carlo methods to approximate the partition function, thereby allowing us to tackle the extra computational burden associated with adding the language model. This approach is theoreti- cally justified and means that the model contin- ues to be both probabilistic and globally optimised. As expected, using a language model dramatically increases translation performance. Our second major contribution is an exploita- tion of syntactic features. By encoding source syn- tax as features allows the model to use, or ignore, this information as it sees fit, thereby avoiding the problems of coverage and sparsity associated with directly incorporating the syntax into the grammar (Huang et al., 2006; Mi et al., 2008). We report on translation gains using this approach. We begin by introducing the synchronous gram- mar approach to SMT in Section 2. In Section 3 we define the parametric form of our model and describe techniques for approximating the intractable space of all translations for a given source sentence. In Section 4 we evaluate the abil- ity of our model to effectively estimate …

Content truncated to 500 words