Bayesian Induction of Bracketing Inversion Transduction Grammars

International Joint Conference on Natural Language Processing, pages 1158–1166, Nagoya, Japan, 14-18 October 2013. Bayesian Induction of Bracketing Inversion Transduction Grammars Markus Saers Dekai Wu Human Language Technology Center Department of Computer Science and Engineering Hong Kong University of Science and Technology {masaers|dekai}@cs.ust.hk Abstract We present a novel approach to learning phrasal inversion transduction grammars via Bayesian MAP (maximum a posteri- ori) or information-theoretic MDL (mini- mum description length) model optimiza- tion so as to incorporate simultaneously the choices of model structure as well as pa- rameters. In comparison to most current SMT approaches, the model learns phrase translation lexicons that (a) do not re- quire enormous amounts of run-time mem- ory, (b) contain significantly less redun- dancy, and (c) provide an obvious basis for generalization to abstract translation schemas. Model structure choice is biased by a description length prior, while pa- rameter choice is driven by data likelihood biased by a parameter prior. The search over possible model structures is made feasible by a novel top-down rule seg- menting heuristic which efficiently incor- porates estimates of the posterior probabil- ities. Since the priors reward model parsi- mony, the learned grammar is very concise and still performs significantly better than the maximum likelihood driven bottom-up rule chunking baseline. 1 Introduction We introduce a minimalist, unsupervised learn- ing model that induces relatively clean, compact phrasal translation lexicons by employing a novel Bayesian approach that attempts to find the maxi- mum a posteriori (MAP) or minimum description length (MDL) model. The approach iteratively segments the rules in a top-down fashion which al- lows for efficient estimation of the model prior and the likelihood of the data given a limited change in the model—it is, in other words, possible to gener- ate a set of possible rule segmentations and com- pare them using the model posteriors or description length. Our new approach differs from most other SMT approaches to unsupervised learning of phrasal translations, which (a) require enormous amounts of run-time memory, (b) contain a high degree of redundancy, and (c) do not provide an ob- vious basis for generalization to abstract trans- lation schemas. The current state-of-the-art in SMT (Koehn et al., 2003; Chiang, 2005) relies on long pipelines of mismatched learning models and heuristics. There is no way for latter stages of the pipeline to recover a mistake of ommission made in an earlier stage, which forces the indi- vidual steps to massively overgenerate hypotheses. This typically manifests as massive redundancy in the phrasal lexicon, which causes significant over- head at run-time. The fact that it is even possi- ble to improve the performance of a phrase-based direct translation system by tossing away most of the learned segmental translations (Johnson et al., 2007) illustrates these deficiencies well. By stay- ing within a single framework throughout train- ing and testing, we do not have to overgenerate hypotheses—instead, we are able to evaluate their effect on the posterior model probability at the time they are proposed during learning. This cuts down the size …

Content truncated to 500 words