Bayesian Learning of Non-Compositional Phrases with Synchronous Parsing

Proceedings of ACL-08: HLT, pages 97–105, Columbus, Ohio, USA, June 2008. c⃝2008 Association for Computational Linguistics Bayesian Learning of Non-compositional Phrases with Synchronous Parsing Hao Zhang Computer Science Department University of Rochester Rochester, NY 14627 zhanghao@cs.rochester.edu Chris Quirk Microsoft Research One Microsoft Way Redmond, WA 98052 USA chrisq@microsoft.com Robert C. Moore Microsoft Research One Microsoft Way Redmond, WA 98052 USA bobmoore@microsoft.com Daniel Gildea Computer Science Department University of Rochester Rochester, NY 14627 gildea@cs.rochester.edu Abstract We combine the strengths of Bayesian mod- eling and synchronous grammar in unsu- pervised learning of basic translation phrase pairs. The structured space of a synchronous grammar is a natural fit for phrase pair proba- bility estimation, though the search space can be prohibitively large. Therefore we explore efficient algorithms for pruning this space that lead to empirically effective results. Incorpo- rating a sparse prior using Variational Bayes, biases the models toward generalizable, parsi- monious parameter sets, leading to significant improvements in word alignment. This pref- erence for sparse solutions together with ef- fective pruning methods forms a phrase align- ment regimen that produces better end-to-end translations than standard word alignment ap- proaches. 1 Introduction Most state-of-the-art statistical machine transla- tion systems are based on large phrase tables ex- tracted from parallel text using word-level align- ments. These word-level alignments are most of- ten obtained using Expectation Maximization on the conditional generative models of Brown et al. (1993) and Vogel et al. (1996). As these word-level align- ment models restrict the word alignment complex- ity by requiring each target word to align to zero or one source words, results are improved by align- ing both source-to-target as well as target-to-source, then heuristically combining these alignments. Fi- nally, the set of phrases consistent with the word alignments are extracted from every sentence pair; these form the basis of the decoding process. While this approach has been very successful, poor word- level alignments are nonetheless a common source of error in machine translation systems. A natural solution to several of these issues is unite the word-level and phrase-level models into one learning procedure. Ideally, such a procedure would remedy the deficiencies of word-level align- ment models, including the strong restrictions on the form of the alignment, and the strong inde- pendence assumption between words. Furthermore it would obviate the need for heuristic combina- tion of word alignments. A unified procedure may also improve the identification of non-compositional phrasal translations, and the attachment decisions for unaligned words. In this direction, Expectation Maximization at the phrase level was proposed by Marcu and Wong (2002), who, however, experienced two major dif- ficulties: computational complexity and controlling overfitting. Computational complexity arises from the exponentially large number of decompositions of a sentence pair into phrase pairs; overfitting is a problem because as EM attempts to maximize the likelihood of its training data, it prefers to directly explain a sentence pair with a single phrase pair. In this paper, we attempt to address these two is- sues in order to apply EM above the …

Content truncated to 500 words