SPEDE: Probabilistic Edit Distance Metrics for MT Evaluation

Proceedings of the 7th Workshop on Statistical Machine Translation, pages 76–83, Montr´eal, Canada, June 7-8, 2012. c⃝2012 Association for Computational Linguistics SPEDE: Probabilistic Edit Distance Metrics for MT Evaluation Mengqiu Wang and Christopher D. Manning Computer Science Department Stanford University Stanford, CA 94305 USA {mengqiu,manning}@cs.stanford.edu Abstract This paper describes Stanford University’s sub- mission to the Shared Evaluation Task of WMT 2012. Our proposed metric (SPEDE) com- putes probabilistic edit distance as predictions of translation quality. We learn weighted edit distance in a probabilistic finite state machine (pFSM) model, where state transitions corre- spond to edit operations. While standard edit distance models cannot capture long-distance word swapping or cross alignments, we rectify these shortcomings using a novel pushdown automaton extension of the pFSM model. Our models are trained in a regression framework, and can easily incorporate a rich set of linguis- tic features. Evaluated on two different pre- diction tasks across a diverse set of datasets, our methods achieve state-of-the-art correla- tion with human judgments. 1 Introduction We describe the Stanford Probabilistic Edit Distance Evaluation (SPEDE) metric, which makes predic- tions of translation quality by computing weighted edit distance. We model weighted edit distance in a probabilistic finite state machine (pFSM), where state transitions correspond to edit operations. The weights of the edit operations are then automatically learned in a regression framework. One of the ma- jor contributions of this paper is a novel extension of the pFSM model into a probabilistic Pushdown Automaton (pPDA), which enhances traditional edit- distance models with the ability to model phrase shift and word swapping. Furthermore, we give a new log- linear parameterization to the pFSM model, which allows it to easily incorporate rich linguistic features. We conducted extensive experiments on a di- verse set of standard evaluation data sets (NIST OpenMT06, 08; WMT06, 07, 08). Our models achieve or surpass state-of-the-art results on all test sets. 2 Related Work Research in automatic machine translation (MT) eval- uation metrics has been a key driving force behind the recent advances of statistical machine transla- tion (SMT) systems. The early seminal work on automatic MT metrics (e.g., BLEU and NIST) is largely based on n-gram matches (Papineni et al., 2002; Doddington, 2002). Despite their simplicity, these measures have shown good correlation with hu- man judgments, and enabled large-scale evaluations across many different MT systems, without incurring the huge labor cost of human evaluation (Callison- Burch et al. (2009; 2010; 2011), inter alia). Later metrics that move beyond n-grams achieve higher accuracy and improved robustness from re- sources like WordNet synonyms (Miller et al., 1990), paraphrasing (Zhou et al., 2006; Snover et al., 2009; Denkowski and Lavie, 2010), and syntactic parse structures (Liu et al., 2005; Owczarzak et al., 2008; He et al., 2010). But a common problem in these metrics is they typically resort to ad-hoc tuning meth- ods instead of principled approaches to incorporate linguistic features. Recent models use linear or SVM regression and train them against human judg- ments to automatic learn feature weights, and …

Content truncated to 500 words