Further Meta-Evaluation of Machine Translation

Proceedings of the Third Workshop on Statistical Machine Translation, pages 70–106, Columbus, Ohio, USA, June 2008. c⃝2008 Association for Computational Linguistics Further Meta-Evaluation of Machine Translation Chris Callison-Burch Johns Hopkins University ccb cs jhu edu Cameron Fordyce camfordyce gmail com Philipp Koehn University of Edinburgh pkoehn inf ed ac uk Christof Monz Queen Mary, University of London christof dcs qmul ac uk Josh Schroeder University of Edinburgh j schroeder ed ac uk Abstract This paper analyzes the translation qual- ity of machine translation systems for 10 language pairs translating between Czech, English, French, German, Hungarian, and Spanish. We report the translation quality of over 30 diverse translation systems based on a large-scale manual evaluation involv- ing hundreds of hours of effort. We use the human judgments of the systems to analyze automatic evaluation metrics for translation quality, and we report the strength of the cor- relation with human judgments at both the system-level and at the sentence-level. We validate our manual evaluation methodol- ogy by measuring intra- and inter-annotator agreement, and collecting timing informa- tion. 1 Introduction This paper presents the results the shared tasks of the 2008 ACL Workshop on Statistical Machine Trans- lation, which builds on two past workshops (Koehn and Monz, 2006; Callison-Burch et al., 2007). There were two shared tasks this year: a translation task which evaluated translation between 10 pairs of Eu- ropean languages, and an evaluation task which ex- amines automatic evaluation metrics. There were a number of differences between this year’s workshop and last year’s workshop: • Test set selection – Instead of creating our test set by reserving a portion of the training data, we instead hired translators to translate a set of newspaper articles from a number of different sources. This out-of-domain test set contrasts with the in-domain Europarl test set. • New language pairs – We evaluated the qual- ity of Hungarian-English machine translation. Hungarian is a challenging language because it is agglutinative, has many cases and verb con- jugations, and has freer word order. German- Spanish was our first language pair that did not include English, but was not manually evalu- ated since it attracted minimal participation. • System combination – Saarland University entered a system combination over a number of rule-based MT systems, and provided their output, which were also treated as fully fledged entries in the manual evaluation. Three addi- tional groups were invited to apply their system combination algorithms to all systems. • Refined manual evaluation – Because last year’s study indicated that fluency and ade- quacy judgments were slow and unreliable, we dropped them from manual evaluation. We re- placed them with yes/no judgments about the acceptability of translations of shorter phrases. • Sentence-level correlation – In addition to measuring the correlation of automatic evalu- ation metrics with human judgments at the sys- tem level, we also measured how consistent they were with the human rankings of individ- ual sentences. The remainder of this paper is organized as fol- lows: Section 2 gives an overview …

Content truncated to 500 words