Fluency, Adequacy, or HTER? Exploring Different Human Judgments with a Tunable MT Metric

Proceedings of the Fourth Workshop on Statistical Machine Translation , pages 259–268, Athens, Greece, 30 March – 31 March 2009. c⃝2009 Association for Computational Linguistics Fluency, Adequacy, or HTER? Exploring Different Human Judgments with a Tunable MT Metric Matthew Snover§, Nitin Madnani§, Bonnie J. Dorr§ † & Richard Schwartz† ‡ §Laboratory for Computational Linguistics and Information Processing §Institute for Advanced Computer Studies §University of Maryland, College Park †Human Language Technology Center of Excellence ‡BBN Technologies {snover,nmadnani,bonnie}@umiacs.umd.edu schwartz@bbn.com Abstract Automatic Machine Translation (MT) evaluation metrics have traditionally been evaluated by the correlation of the scores they assign to MT output with human judgments of translation performance. Different types of human judgments, such as Fluency, Adequacy, and HTER, mea- sure varying aspects of MT performance that can be captured by automatic MT metrics. We explore these differences through the use of a new tunable MT met- ric: TER-Plus, which extends the Transla- tion Edit Rate evaluation metric with tun- able parameters and the incorporation of morphology, synonymy and paraphrases. TER-Plus was shown to be one of the top metrics in NIST’s Metrics MATR 2008 Challenge, having the highest aver- age rank in terms of Pearson and Spear- man correlation. Optimizing TER-Plus to different types of human judgments yields significantly improved correlations and meaningful changes in the weight of different types of edits, demonstrating sig- nificant differences between the types of human judgments. 1 Introduction Since the introduction of the BLEU metric (Pa- pineni et al., 2002), statistical MT systems have moved away from human evaluation of their per- formance and towards rapid evaluation using au- tomatic metrics. These automatic metrics are themselves evaluated by their ability to generate scores for MT output that correlate well with hu- man judgments of translation quality. Numer- ous methods of judging MT output by humans have been used, including Fluency, Adequacy, and, more recently, Human-mediated Translation Edit Rate (HTER) (Snover et al., 2006). Fluency measures whether a translation is fluent, regard- less of the correct meaning, while Adequacy mea- sures whether the translation conveys the correct meaning, even if the translation is not fully flu- ent. Fluency and Adequacy are frequently mea- sured together on a discrete 5 or 7 point scale, with their average being used as a single score of translation quality. HTER is a more complex and semi-automatic measure in which humans do not score translations directly, but rather generate a new reference translation that is closer to the MT output but retains the fluency and meaning of the original reference. This new targeted refer- ence is then used as the reference translation when scoring the MT output using Translation Edit Rate (TER) (Snover et al., 2006) or when used with other automatic metrics such as BLEU or ME- TEOR (Banerjee and Lavie, 2005). One of the difficulties in the creation of targeted references is a further requirement that the annotator attempt to minimize the number of edits, as measured by TER, between the MT output and the targeted ref- erence, creating the …

Content truncated to 500 words