A Systematic Bayesian Treatment of the IBM Alignment Models

Proceedings of NAACL-HLT 2013, pages 969–977, Atlanta, Georgia, 9–14 June 2013. c⃝2013 Association for Computational Linguistics A Systematic Bayesian Treatment of the IBM Alignment Models Yarin Gal Department of Engineering University of Cambridge Cambridge, CB2 1PZ, United Kingdom yg279@cam.ac.uk Phil Blunsom Department of Computer Science University of Oxford Oxford, OX1 3QD, United Kingdom Phil.Blunsom@cs.ox.ac.uk Abstract The dominant yet ageing IBM and HMM word alignment models underpin most popular Statistical Machine implementations in use today.Translation Though beset by the limitations of implausible independence assumptions, intractable optimisation problems, and an excess of tunable parameters, these models provide a scalable and reliable starting point for inducing translation systems. In this paper we build upon this venerable base by recasting these models in the non-parametric Bayesian framework. By replacing the categorical distributions at their core with hierarchical Pitman-Yor processes, and through the use of collapsed Gibbs sampling, we provide a more flexible formulation and sidestep the original heuristic optimisation techniques. The resulting models are highly extendible, naturally permitting the introduction of phrasal dependencies. We present extensive experimental results showing improvements in both AER and BLEU when benchmarked against Giza++, including significant improvements over IBM model 4. 1 Introduction The IBM and HMM word alignment models (Brown et al., 1993; Vogel et al., 1996) have underpinned the majority of statistical machine translation systems for almost twenty years. The key attraction of these models is their principled probabilistic formulation, and the existence of (mostly) tractable algorithms for their training. The dominant Giza++ implementation of the IBM models (Och and Ney, 2003) employs a variety of exact and approximate EM algorithms to optimise categorical alignment distributions. While effective, this parametric approach results in a significant number of parameters to be tuned and intractable summations over the space of alignments for models 3 and 4. Giza++ hides the hyper- parameters with defaults and approximates the intractable expectations using restricted alignment neighbourhoods. However this approach was shown to often return alignments with probabilities well below the true maxima (Ravi and Knight, 2010). To overcome perceived limitations with the word based and non-syntactic nature of the IBM models many alternative approaches to word alignment have been proposed (e.g. (DeNero et al., 2008; Cohn and Blunsom, 2009; Levenberg et al., 2012)). While interesting results have been reported, these alterna- tives have failed to dislodge the IBM approach. In this paper we proposed to retain the original generative stories of the IBM models, while replacing the inflexible categorical distributions with hierarchical Pitman-Yor (PY) processes – a mathematical tool which has been successfully applied to a range of language tasks (Teh, 2006b; Goldwater et al., 2006; Blunsom and Cohn, 2011). In the context of language modelling, the hierarchical PY process was shown to roughly correspond to interpolated Kneser-Ney (Kneser and Ney, 1995; Teh, 2006a). The key contribution of the hierarchical PY formulation is that it provides a principle probabilistic framework that easily extends to latent variable models, such as those used 969 for alignment, for which a Kneser-Ney formulation is unclear. While Bayesian …

Content truncated to 500 words