Probabilistic Finite State Machines for Regression-based MT Evaluation

Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 984–994, Jeju Island, Korea, 12–14 July 2012. c⃝2012 Association for Computational Linguistics Probabilistic Finite State Machines for Regression-based MT Evaluation Mengqiu Wang and Christopher D. Manning Computer Science Department Stanford University Stanford, CA 94305 USA {mengqiu,manning}@cs.stanford.edu Abstract Accurate and robust metrics for automatic eval- uation are key to the development of statistical machine translation (MT) systems. We first introduce a new regression model that uses a probabilistic finite state machine (pFSM) to compute weighted edit distance as predictions of translation quality. We also propose a novel pushdown automaton extension of the pFSM model for modeling word swapping and cross alignments that cannot be captured by stan- dard edit distance models. Our models can eas- ily incorporate a rich set of linguistic features, and automatically learn their weights, elimi- nating the need for ad-hoc parameter tuning. Our methods achieve state-of-the-art correla- tion with human judgments on two different prediction tasks across a diverse set of standard evaluations (NIST OpenMT06,08; WMT06- 08). 1 Introduction Research in automatic machine translation (MT) eval- uation metrics has been a key driving force behind the recent advances of statistical machine transla- tion (SMT) systems. The early seminal work on automatic MT metrics (e.g., BLEU and NIST) is largely based on n-gram matches (Papineni et al., 2002; Doddington, 2002). Despite their simplicity, these measures have shown good correlation with hu- man judgments, and enabled large-scale evaluations across many different MT systems, without incurring the huge labor cost of human evaluation (Callison- Burch et al. (2009; 2010; 2011), inter alia). Recent studies have also confirmed that tuning MT systems against better MT metrics — using algorithms like MERT (Och, 2003) — leads to better system perfor- mance (He and Way, 2009; Liu et al., 2011). Later metrics that move beyond n-grams achieve higher accuracy and improved robustness from re- sources like WordNet synonyms (Miller et al., 1990), and paraphrasing (Snover et al., 2009; Denkowski and Lavie, 2010). But a common problem in these metrics is they typically resort to ad-hoc tuning methods instead of principled approaches to incor- porate linguistic features. Recent models use linear or SVM regression and train them against human judgments to automatic learn feature weights, and have shown state-of-the-art correlation with human judgments (Kulesza and Shieber, 2004; Albrecht and Hwa, 2007a; Albrecht and Hwa, 2007b; Sun et al., 2008; Pado et al., 2009). The drawback, however, is they rely on time-consuming preprocessing mod- ules to extract linguistic features (e.g., a full end-to- end textual entailment system was needed in Pado et al. (2009)), which severely limits their practical use. Furthermore, these models employ a large number of features (on the order of hundreds), and conse- quently make the model predictions opaque and hard to analyze. In this paper, we propose a simple yet powerful probabilistic Finite State Machine (pFSM) for the task of MT evaluation. It is built on the backbone of weighted edit distance …

Content truncated to 500 words