Regularizing Mono- and Bi-Word Models for Word Alignment

Proceedings of the 5th International Joint Conference on Natural Language Processing, pages 1313–1322, Chiang Mai, Thailand, November 8 – 13, 2011. c⃝2011 AFNLP Regularizing Mono- and Bi-Word Models for Word Alignment Thomas Schoenemann Lund University, Sweden Abstract Conditional probabilistic models for word alignment are popular due to the elegant way of handling them in the training stage. However, they have weaknesses such as garbage collection and scale poorly be- yond single word based models (DeNero et al., 2006): not all parameters should ac- tually be used. To alleviate the problem, in this paper we explore regularity terms that penalize the used parameters. They share the advan- tages of the standard training in that itera- tive schemes decompose over the sentence pairs. We explore the models IBM-1 and HMM, then generalize to models we term Bi-word models, where each target word can be aligned to up to two source words. We give two optimization strategies for the arising tasks, using EM and projected gra- dient descent. While both are well-known, to our knowledge they have never been compared experimentally for the task of word alignment. As a side-effect, we show that, against common belief, for paramet- ric HMMs the M-step is not solved by re- normalizing expectations. We demonstrate that the regularity terms improve on the f-measures of the standard HMMs and that they improve translation quality. 1 Introduction State-of-the art approaches for word alignment are based on probabilistic models. They can be split into joint models (Melamed, 2000; Marcu and Wong, 2002) and conditional models (Brown et al., 1993; Vogel et al., 1996; Wang and Waibel, 1998; Toutanova et al., 2002; Sumita et al., 2004; Deng and Byrne, 2005; Fraser and Marcu, 2007a). While in early works the underlying basic entity was a single word, today’s advanced approaches build on sequences of words, called phrases. For joint models the advanced models are stand- alone approaches (Marcu and Wong, 2002). How- ever, these models are computationally hard to handle, which frequently results in maximum ap- proximations being made. This is different for the conditional models, which are easier to handle but where most approaches are based on initializ- ing from single-word based models (Brown et al., 1993; Vogel et al., 1996; Al-Onaizan et al., 1999). However, the recent work of Mauser et al. (2009) deals with pairs of source words and is trained without considering single word based models. In this paper we much generalize on this work, considering a class of models we term Bi-word models. We consider a variant of (Mauser et al., 2009) which we call Bi-1, then proceed to derive a Bi-HMM. Our main focus is however on regu- larizing such models. We first address known con- ditional models called single-word based models, focusing on a weakness known as garbage collec- tion. We show that this weakness can be alleviated by adding an entry to every dictionary distribution as well as adding a regularity term (a weighted L1 norm). Afterwards we generalize this idea to Bi- word …

Content truncated to 500 words