Training Nondeficient Variants of IBM-3 and IBM-4 for Word Alignment

Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pages 22–31, Sofia, Bulgaria, August 4-9 2013. c⃝2013 Association for Computational Linguistics Training Nondeficient Variants of IBM-3 and IBM-4 for Word Alignment Thomas Schoenemann Heinrich-Heine-Universit¨at D¨usseldorf, Germany Universit¨atsstr. 1 40225 D¨usseldorf, Germany Abstract We derive variants of the fertility based models IBM-3 and IBM-4 that, while maintaining their zero and first order pa- rameters, are nondeficient. Subsequently, we proceed to derive a method to com- pute a likely alignment and its neighbors as well as give a solution of EM training. The arising M-step energies are non-trivial and handled via projected gradient ascent. Our evaluation on gold alignments shows substantial improvements (in weighted F- measure) for the IBM-3. For the IBM- 4 there are no consistent improvements. Training the nondeficient IBM-5 in the regular way gives surprisingly good re- sults. Using the resulting alignments for phrase- based translation systems offers no clear insights w.r.t. BLEU scores. 1 Introduction While most people think of the translation and word alignment models IBM-3 and IBM-4 as in- herently deficient models (i.e. models that assign non-zero probability mass to impossible events), in this paper we derive nondeficient variants main- taining their zero order (IBM-3) and first order (IBM-4) parameters. This is possible as IBM-3 and IBM-4 are very special cases of general log- linear models: they are properly derived by the chain rule of probabilities. Deficiency is only in- troduced by ignoring a part of the history to be conditioned on in the individual factors of the chain rule factorization. While at first glance this seems necessary to obtain zero and first order de- Figure 1: Plot of the negative log. likelihoods (the quantity to be minimized) arising in training deficient and nondeficient models (for Europarl German | English, training scheme 15H53545). 1/3/4=IBM-1/3/4, H=HMM, T=Transfer iteration. The curves are identical up to iteration 11. Iteration 11 shows that merely 5.14% of the (HMM) probability mass are covered by the Viterbi alignment and its neighbors. With deficient models (and deficient empty words) the final neg- ative log likelihood is higher than the initial HMM one, with nondeficient models it is lower than for the HMM, as it should be for a better model. pendencies, we show that with proper renormal- ization all factors can be made nondeficient. Having introduced the model variants, we pro- ceed to derive a hillclimbing method to compute a likely alignment (ideally the Viterbi alignment) and its neighbors. As for the deficient models, this plays an important role in the E-step of the sub- sequently derived expectation maximization (EM) training scheme. As usual, expectations in EM are approximated, but we now also get non-trivial M- step energies. We deal with these via projected gradient ascent. 22 The downside of our method is its resource con- sumption, but still we present results on corpora with 100.000 sentence pairs. The source code of this project is available in our word alignment soft- ware RegAligner1, version 1.2 and later. Figure 1 gives …

Content truncated to 500 words