Improving IBM Word Alignment Model 1

Improving IBM Word-Alignment Model 1 Robert C. MOORE Microsoft Research One Microsoft Way Redmond, WA 90052 USA bobmoore@microsoft.com Abstract We investigate a number of simple methods for improving the word-alignment accuracy of IBM Model 1. We demonstrate reduction in alignment error rate of approximately 30% resulting from (1) giving extra weight to the probability of alignment to the null word, (2) smoothing probability esti- mates for rare words, and (3) using a simple heuris- tic estimation method to initialize, or replace, EM training of model parameters. 1 Introduction IBM Model 1 (Brown et al., 1993a) is a word- alignment model that is widely used in working with parallel bilingual corpora. It was originally developed to provide reasonable initial parameter estimates for more complex word-alignment mod- els, but it has subsequently found a host of ad- ditional uses. Among the applications of Model 1 are segmenting long sentences into subsentental units for improved word alignment (Nevado et al., 2003), extracting parallel sentences from compara- ble corpora (Munteanu et al., 2004), bilingual sen- tence alignment (Moore, 2002), aligning syntactic- tree fragments (Ding et al., 2003), and estimating phrase translation probabilities (Venugopal et al., 2003). Furthermore, at the 2003 Johns Hopkins summer workshop on statistical machine transla- tion, a large number of features were tested to dis- cover which ones could improve a state-of-the-art translation system, and the only feature that pro- duced a “truly significant improvement” was the Model 1 score (Och et al., 2004). Despite the fact that IBM Model 1 is so widely used, essentially no attention seems to have been paid to whether it is possible to improve on the stan- dard Expectation-Maximization (EM) procedure for estimating its parameters. This may be due in part to the fact that Brown et al. (1993a) proved that the log-likelihood objective function for Model 1 is a strictly concave function of the model parameters, so that it has a unique local maximum. This, in turn, means that EM training will converge to that max- imum from any starting point in which none of the initial parameter values is zero. If one equates opti- mum parameter estimation with finding the global maximum for the likelihood of the training data, then this result would seem to show no improve- ment is possible. However, in virtually every application of statisti- cal techniques in natural-language processing, max- imizing the likelihood of the training data causes overfitting, resulting in lower task performance than some other estimates for the model parameters. This is implicitly recognized in the widespread adoption of early stopping in estimating the parameters of Model 1. Brown et al. (1993a) stopped after only one iteration of EM in using Model 1 to initialize their Model 2, and Och and Ney (2003) stop af- ter five iterations in using Model 1 to initialize the HMM word-alignment model. Both of these are far short of convergence to the maximum likelihood es- timates for the model parameters. We have identified at least two ways in which the standard …

Content truncated to 500 words