(Meta-) Evaluation of Machine Translation

Proceedings of the Second Workshop on Statistical Machine Translation, pages 136–158, Prague, June 2007. c⃝2007 Association for Computational Linguistics (Meta-) Evaluation of Machine Translation Chris Callison-Burch Johns Hopkins University ccb clsp jhu edu Cameron Fordyce CELCT fordyce celct it Philipp Koehn University of Edinburgh pkoehn inf ed ac uk Christof Monz Queen Mary, University of London christof dcs qmul ac uk Josh Schroeder University of Edinburgh j schroeder ed ac uk Abstract This paper evaluates the translation quality of machine translation systems for 8 lan- guage pairs: translating French, German, Spanish, and Czech to English and back. We carried out an extensive human evalua- tion which allowed us not only to rank the different MT systems, but also to perform higher-level analysis of the evaluation pro- cess. We measured timing and intra- and inter-annotator agreement for three types of subjective evaluation. We measured the cor- relation of automatic evaluation metrics with human judgments. This meta-evaluation re- veals surprising facts about the most com- monly used methodologies. 1 Introduction This paper presents the results for the shared trans- lation task of the 2007 ACL Workshop on Statistical Machine Translation. The goals of this paper are twofold: First, we evaluate the shared task entries in order to determine which systems produce trans- lations with the highest quality. Second, we analyze the evaluation measures themselves in order to try to determine “best practices” when evaluating machine translation research. Previous ACL Workshops on Machine Transla- tion were more limited in scope (Koehn and Monz, 2005; Koehn and Monz, 2006). The 2005 workshop evaluated translation quality only in terms of Bleu score. The 2006 workshop additionally included a limited manual evaluation in the style of NIST ma- chine translation evaluation workshop. Here we ap- ply eleven different automatic evaluation metrics, and conduct three different types of manual evalu- ation. Beyond examining the quality of translations pro- duced by various systems, we were interested in ex- amining the following questions about evaluation methodologies: How consistent are people when they judge translation quality? To what extent do they agree with other annotators? Can we im- prove human evaluation? Which automatic evalu- ation metrics correlate most strongly with human judgments of translation quality? This paper is organized as follows: • Section 2 gives an overview of the shared task. It describes the training and test data, reviews the baseline system, and lists the groups that participated in the task. • Section 3 describes the manual evaluation. We performed three types of evaluation: scoring with five point scales, relative ranking of trans- lations of sentences, and ranking of translations of phrases. • Section 4 lists the eleven different automatic evaluation metrics which were also used to score the shared task submissions. • Section 5 presents the results of the shared task, giving scores for each of the systems in each of the different conditions. • Section 6 provides an evaluation of the dif- ferent types of evaluation, giving intra- and 136 inter-annotator agreement figures for the man- ual evaluation, …

Content truncated to 500 words