A Simple, Fast, and Effective Reparameterization of IBM Model 2

Proceedings of NAACL-HLT 2013, pages 644–648, Atlanta, Georgia, 9–14 June 2013. c⃝2013 Association for Computational Linguistics A Simple, Fast, and Effective Reparameterization of IBM Model 2 Chris Dyer Victor Chahuneau Noah A. Smith Language Technologies Institute Carnegie Mellon University Pittsburgh, PA 15213, USA {cdyer,vchahune,nasmith}@cs.cmu.edu Abstract We present a simple log-linear reparame- terization of IBM Model 2 that overcomes problems arising from Model 1’s strong assumptions and Model 2’s overparame- terization. Efficient inference, likelihood evaluation, and parameter estimation algo- rithms are provided. Training the model is consistently ten times faster than Model 4. On three large-scale translation tasks, systems built using our alignment model outperform IBM Model 4. An open-source implementation of the align- ment model described in this paper is available from http://github.com/clab/fast align . 1 Introduction Word alignment is a fundamental problem in statis- tical machine translation. While the search for more sophisticated models that provide more nuanced ex- planations of parallel corpora is a key research activ- ity, simple and effective models that scale well are also important. These play a crucial role in many scenarios such as parallel data mining and rapid large scale experimentation, and as subcomponents of other models or training and inference algorithms. For these reasons, IBM Models 1 and 2, which sup- port exact inference in time Θ(|f| · |e|), continue to be widely used. This paper argues that both of these models are suboptimal, even in the space of models that per- mit such computationally cheap inference. Model 1 assumes all alignment structures are uniformly likely (a problematic assumption, particularly for frequent word types), and Model 2 is vastly overpa- rameterized, making it prone to degenerate behav- ior on account of overfitting.1 We present a simple log-linear reparameterization of Model 2 that avoids both problems (§2). While inference in log-linear models is generally computationally more expen- sive than in their multinomial counterparts, we show how the quantities needed for alignment inference, likelihood evaluation, and parameter estimation us- ing EM and related methods can be computed using two simple algebraic identities (§3), thereby defus- ing this objection. We provide results showing our model is an order of magnitude faster to train than Model 4, that it requires no staged initialization, and that it produces alignments that lead to significantly better translation quality on downstream translation tasks (§4). 2 Model Our model is a variation of the lexical translation models proposed by Brown et al. (1993). Lexical translation works as follows. Given a source sen- tence f with length n, first generate the length of the target sentence, m. Next, generate an alignment, a = ⟨a1, a2, . . . , am⟩, that indicates which source word (or null token) each target word will be a trans- lation of. Last, generate the m output words, where each ei depends only on fai. The model of alignment configurations we pro- pose is a log-linear reparameterization of Model 2. 1Model 2 has independent parameters for every alignment position, conditioned on the source length, target length, and …

Content truncated to 500 words