Regression for Sentence-Level MT Evaluation with Pseudo References

Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 296–303, Prague, Czech Republic, June 2007. c⃝2007 Association for Computational Linguistics Regression for Sentence-Level MT Evaluation with Pseudo References Joshua S. Albrecht and Rebecca Hwa Department of Computer Science University of Pittsburgh {jsa8,hwa}@cs.pitt.edu Abstract Many automatic evaluation metrics for ma- chine translation (MT) rely on making com- parisons to human translations, a resource that may not always be available. We present a method for developing sentence-level MT evaluation metrics that do not directly rely on human reference translations. Our met- rics are developed using regression learn- ing and are based on a set of weaker indi- cators of fluency and adequacy (pseudo ref- erences). Experimental results suggest that they rival standard reference-based metrics in terms of correlations with human judg- ments on new test instances. 1 Introduction Automatic assessment of translation quality is a challenging problem because the evaluation task, at its core, is based on subjective human judgments. Reference-based metrics such as BLEU (Papineni et al., 2002) have rephrased this subjective task as a somewhat more objective question: how closely does the translation resemble sentences that are known to be good translations for the same source? This approach requires the participation of human translators, who provide the “gold standard” refer- ence sentences. However, keeping humans in the evaluation loop represents a significant expenditure both in terms of time and resources; therefore it is worthwhile to explore ways of reducing the degree of human involvement. To this end, Gamon et al. (2005) proposed a learning-based evaluation metric that does not com- pare against reference translations. Under a learn- ing framework, the input (i.e., the sentence to be evaluated) is represented as a set of features. These are measurements that can be extracted from the in- put sentence (and may be individual metrics them- selves). The learning algorithm combines the fea- tures to form a model (a composite evaluation met- ric) that produces the final score for the input. With- out human references, the features in the model pro- posed by Gamon et al. were primarily language model features and linguistic indicators that could be directly derived from the input sentence alone. Al- though their initial results were not competitive with standard reference-based metrics, their studies sug- gested that a referenceless metric may still provide useful information about translation fluency. How- ever, a potential pitfall is that systems might “game the metric” by producing fluent outputs that are not adequate translations of the source. This paper proposes an alternative approach to evaluate MT outputs without comparing against hu- man references. While our metrics are also trained, our model consists of different features and is trained under a different learning regime. Crucially, our model includes features that capture some no- tions of adequacy by comparing the input against pseudo references: sentences from other MT sys- tems (such as commercial off-the-shelf systems or open sourced research systems). To improve flu- ency judgments, the model also includes features that compare the …

Content truncated to 500 words