A Systematic Analysis of Translation Model Search Spaces

Proceedings of the Fourth Workshop on Statistical Machine Translation , pages 224–232, Athens, Greece, 30 March – 31 March 2009. c⃝2009 Association for Computational Linguistics A Systematic Analysis of Translation Model Search Spaces Michael Auli, Adam Lopez, Hieu Hoang and Philipp Koehn University of Edinburgh 10 Crichton Street Edinburgh, EH8 9AB United Kingdom m.auli@sms.ed.ac.uk, alopez@inf.ed.ac.uk, h.hoang@sms.ed.ac.uk, pkoehn@inf.ed.ac.uk Abstract Translation systems are complex, and most metrics do little to pinpoint causes of error or isolate system differences. We use a simple technique to discover induction errors, which occur when good transla- tions are absent from model search spaces. Our results show that a common prun- ing heuristic drastically increases induc- tion error, and also strongly suggest that the search spaces of phrase-based and hi- erarchical phrase-based models are highly overlapping despite the well known struc- tural differences. 1 Introduction Most empirical work in translation analyzes mod- els and algorithms using BLEU (Papineni et al., 2002) and related metrics. Though such met- rics are useful as sanity checks in iterative sys- tem development, they are less useful as analyti- cal tools. The performance of a translation system depends on the complex interaction of several dif- ferent components. Since metrics assess only out- put, they fail to inform us about the consequences of these interactions, and thus provide no insight into the errors made by a system, or into the de- sign tradeoffs of competing systems. In this work, we show that it is possible to ob- tain such insights by analyzing translation sys- tem components in isolation. We focus on model search spaces (§2), posing a very simple question: Given a model and a sentence pair, does the search space contain the sentence pair? Applying this method to the analysis and comparison of French- English translation using both phrase-based and hierarchical phrase-based systems yields surpris- ing results, which we analyze quantitatively and qualitatively. • First, we analyze the induction error of a model, a measure on the completeness of the search space. We find that low weight phrase translations typically discarded by heuristic pruning nearly triples the number of refer- ence sentences that can be exactly recon- structed by either model (§3). • Second, we find that the high-probability re- gions in the search spaces of phrase-based and hierarchical systems are nearly identical (§4). This means that reported differences be- tween the models are due to their rankings of competing hypotheses, rather than structural differences of the derivations they produce. 2 Models, Search Spaces, and Errors A translation model consists of two distinct ele- ments: an unweighted ruleset, and a parameteri- zation (Lopez, 2008a; 2009). A ruleset licenses the steps by which a source string f1...fI may be rewritten as a target string e1...eJ. A parameter- ization defines a weight function over every se- quence of rule applications. In a phrase-based model, the ruleset is simply the unweighted phrase table, where each phrase pair fi...fi′/ej...ej′ states that phrase fi...fi′ in the source can be rewritten as ej...ej′ in the tar- get. The …

Content truncated to 500 words