Findings of the 2010 Joint Workshop on Statistical Machine Translation and Metrics for Machine Translation

Proceedings of the Joint 5th Workshop on Statistical Machine Translation and MetricsMATR, pages 17–53, Uppsala, Sweden, 15-16 July 2010. c⃝2010 Association for Computational Linguistics Findings of the 2010 Joint Workshop on Statistical Machine Translation and Metrics for Machine Translation Chris Callison-Burch Johns Hopkins University ccb@cs.jhu.edu Philipp Koehn University of Edinburgh pkoehn@inf.ed.ac.uk Christof Monz University of Amsterdam c.monz@uva.nl Kay Peterson and Mark Przybocki National Institute of Standards and Technology kay.peterson,mark.przybocki@nist.gov Omar F. Zaidan Johns Hopkins University ozaidan@cs.jhu.edu Abstract This paper presents the results of the WMT10 and MetricsMATR10 shared tasks,1 which included a translation task, a system combination task, and an eval- uation task. We conducted a large-scale manual evaluation of 104 machine trans- lation systems and 41 system combina- tion entries. We used the ranking of these systems to measure how strongly auto- matic metrics correlate with human judg- ments of translation quality for 26 metrics. This year we also investigated increasing the number of human judgments by hiring non-expert annotators through Amazon’s Mechanical Turk. 1 Introduction This paper presents the results of the shared tasks of the joint Workshop on statistical Ma- chine Translation (WMT) and Metrics for MA- chine TRanslation (MetricsMATR), which was held at ACL 2010. This builds on four previ- ous WMT workshops (Koehn and Monz, 2006; Callison-Burch et al., 2007; Callison-Burch et al., 2008; Callison-Burch et al., 2009), and one pre- vious MetricsMATR meeting (Przybocki et al., 2008). There were three shared tasks this year: a translation task between English and four other European languages, a task to combine the out- put of multiple machine translation systems, and a task to predict human judgments of translation quality using automatic evaluation metrics. The 1The MetricsMATR analysis was not complete in time for the publication deadline. An updated version of paper will be made available on http://statmt.org/wmt10/ prior to July 15, 2010. performance on each of these shared task was de- termined after a comprehensive human evaluation. There were a number of differences between this year’s workshop and last year’s workshop: • Non-expert judgments – In addition to hav- ing shared task participants judge translation quality, we also collected judgments from non-expert annotators hired through Ama- zon’s Mechanical Turk. By collecting a large number of judgments we hope to reduce the burden on shared task participants, and to in- crease the statistical significance of our find- ings. We discuss the feasibility of using non- experts evaluators, by analyzing the cost, vol- ume and quality of non-expert annotations. • Clearer results for system combination – This year we excluded Google translations from the systems used in system combina- tion. In last year’s evaluation, the large mar- gin between Google and many of the other systems meant that it was hard to improve on when combining systems. This year, the sys- tem combinations perform better than their component systems more often than last year. • Fewer rule-based systems – This year there were fewer rule-based systems submitted. In past years, University of Saarland compiled a large set of outputs …

Content truncated to 500 words