Why Generative Phrase Models Underperform Surface Heuristics

Proceedings of the Workshop on Statistical Machine Translation, pages 31–38, New York City, June 2006. c⃝2006 Association for Computational Linguistics Why Generative Phrase Models Underperform Surface Heuristics John DeNero, Dan Gillick, James Zhang, Dan Klein Department of Electrical Engineering and Computer Science University of California, Berkeley Berkeley, CA 94705 {denero, dgillick, jyzhang, klein}@eecs.berkeley.edu Abstract We investigate why weights from generative mod- els underperform heuristic estimates in phrase- based machine translation. We first propose a sim- ple generative, phrase-based model and verify that its estimates are inferior to those given by surface statistics. The performance gap stems primarily from the addition of a hidden segmentation vari- able, which increases the capacity for overfitting during maximum likelihood training with EM. In particular, while word level models benefit greatly from re-estimation, phrase-level models do not: the crucial difference is that distinct word alignments cannot all be correct, while distinct segmentations can. Alternate segmentations rather than alternate alignments compete, resulting in increased deter- minization of the phrase table, decreased general- ization, and decreased final BLEU score. We also show that interpolation of the two methods can re- sult in a modest increase in BLEU score. 1 Introduction At the core of a phrase-based statistical machine translation system is a phrase table containing pairs of source and target language phrases, each weighted by a conditional translation probability. Koehn et al. (2003a) showed that translation qual- ity is very sensitive to how this table is extracted from the training data. One particularly surprising result is that a simple heuristic extraction algorithm based on surface statistics of a word-aligned training set outperformed the phrase-based generative model proposed by Marcu and Wong (2002). This result is surprising in light of the reverse sit- uation for word-based statistical translation. Specif- ically, in the task of word alignment, heuristic ap- proaches such as the Dice coefficient consistently underperform their re-estimated counterparts, such as the IBM word alignment models (Brown et al., 1993). This well-known result is unsurprising: re- estimation introduces an element of competition into the learning process. The key virtue of competition in word alignment is that, to a first approximation, only one source word should generate each target word. If a good alignment for a word token is found, other plausible alignments are explained away and should be discounted as incorrect for that token. As we show in this paper, this effect does not pre- vail for phrase-level alignments. The central differ- ence is that phrase-based models, such as the ones presented in section 2 or Marcu and Wong (2002), contain an element of segmentation. That is, they do not merely learn correspondences between phrases, but also segmentations of the source and target sen- tences. However, while it is reasonable to sup- pose that if one alignment is right, others must be wrong, the situation is more complex for segmenta- tions. For example, if one segmentation subsumes another, they are not necessarily incompatible: both may be equally valid. While in some cases, such as idiomatic vs. literal translations, …

Content truncated to 500 words