Findings of the 2009 Workshop on Statistical Machine Translation

Proceedings of the Fourth Workshop on Statistical Machine Translation , pages 1–28, Athens, Greece, 30 March – 31 March 2009. c⃝2009 Association for Computational Linguistics Findings of the 2009 Workshop on Statistical Machine Translation Chris Callison-Burch Johns Hopkins University ccb cs jhu edu Philipp Koehn University of Edinburgh pkoehn inf ed ac uk Christof Monz University of Amsterdam christof science uva nl Josh Schroeder University of Edinburgh j schroeder ed ac uk Abstract This paper presents the results of the WMT09 shared tasks, which included a translation task, a system combination task, and an evaluation task. We con- ducted a large-scale manual evaluation of 87 machine translation systems and 22 system combination entries. We used the ranking of these systems to measure how strongly automatic metrics correlate with human judgments of translation quality, for more than 20 metrics. We present a new evaluation technique whereby system output is edited and judged for correctness. 1 Introduction This paper presents the results of the shared tasks of the 2009 EACL Workshop on Statistical Ma- chine Translation, which builds on three previ- ous workshops (Koehn and Monz, 2006; Callison- Burch et al., 2007; Callison-Burch et al., 2008). There were three shared tasks this year: a transla- tion task between English and five other European languages, a task to combine the output of multiple machine translation systems, and a task to predict human judgments of translation quality using au- tomatic evaluation metrics. The performance on each of these shared task was determined after a comprehensive human evaluation. There were a number of differences between this year’s workshop and last year’s workshop: • Larger training sets – In addition to annual increases in the Europarl corpus, we released a French-English parallel corpus verging on 1 billion words. We also provided large mono- lingual training sets for better language mod- eling of the news translation task. • Reduced number of conditions – Previ- ous workshops had many conditions: 10 language pairs, both in-domain and out-of- domain translation, and three types of man- ual evaluation. This year we eliminated the in-domain Europarl test set and defined sentence-level ranking as the primary type of manual evaluation. • Editing to evaluate translation quality – Beyond ranking the output of translation sys- tems, we evaluated translation quality by hav- ing people edit the output of systems. Later, we asked annotators to judge whether those edited translations were correct when shown the source and reference translation. The primary objectives of this workshop are to evaluate the state of the art in machine transla- tion, to disseminate common test sets and pub- lic training data with published performance num- bers, and to refine evaluation methodologies for machine translation. All of the data, translations, and human judgments produced for our workshop are publicly available.1 We hope they form a valuable resource for research into statistical ma- chine translation, system combination, and auto- matic evaluation of translation quality. 2 Overview of the shared translation and system combination tasks The workshop examined translation between En- …

Content truncated to 500 words