The Role of Pseudo References in MT Evaluation

Proceedings of the Third Workshop on Statistical Machine Translation, pages 187–190, Columbus, Ohio, USA, June 2008. c⃝2008 Association for Computational Linguistics The Role of Pseudo References in MT Evaluation Joshua S. Albrecht and Rebecca Hwa Department of Computer Science University of Pittsburgh {jsa8,hwa}@cs.pitt.edu Abstract Previous studies have shown automatic evalu- ation metrics to be more reliable when com- pared against many human translations. How- ever, multiple human references may not al- ways be available. It is more common to have only a single human reference (extracted from parallel texts) or no reference at all. Our ear- lier work suggested that one way to address this problem is to train a metric to evaluate a sentence by comparing it against pseudo refer- ences, or imperfect “references” produced by off-the-shelf MT systems. In this paper, we further examine the approach both in terms of the training methodology and in terms of the role of the human and pseudo references. Our expanded experiments show that the approach generalizes well across multiple years and dif- ferent source languages. 1 Introduction Standard automatic metrics are reference-based; that is, they compare system-produced translations against human-translated references produced for the same source. Since there is usually no single best way to translate a sentence, each MT output should be compared against many references. On the other hand, creating multiple human references is itself a costly process. For many naturally occur- ring datasets (e.g., parallel corpora) only a single ref- erence is readily available. The focus of this work is on developing auto- matic metrics for sentence-level evaluation with at most one human reference. One way to supple- ment the single human reference is to use pseudo references, or sentences produced by off-the-shelf MT systems, as stand-ins for human references. However, since pseudo references may be imperfect translations themselves, the comparisons cannot be fully trusted. Previously, we have taken a learning- based approach to develop a composite metric that combines measurements taken from multiple pseudo references (Albrecht and Hwa, 2007). Experimental results suggested the approach to be promising; but those studies did not consider how well the metric might generalize across multiple years and different languages. In this paper, we investigate the appli- cability of the pseudo-reference metrics under these more general conditions. Using the WMT06 Workshop shared-task re- sults (Koehn and Monz, 2006) as training exam- ples, we train a metric that evaluates new sentences by comparing them against pseudo references pro- duced by three off-the-shelf MT systems. We ap- ply the learned metric to sentences from the WMT07 shared-task (Callison-Burch et al., 2007b) and com- pare the metric’s predictions against human judg- ments. We find that additional pseudo references improve correlations for automatic metrics. 2 Background The ideal evaluation metric reports an accurate dis- tance between an input instance and its gold stan- dard, but even when comparing against imperfect standards, the measured distances may still convey some useful information – they may help to trian- gulate the input’s position relative to the true gold standard. …

Content truncated to 500 words