Findings of the 2012 Workshop on Statistical Machine Translation

Proceedings of the 7th Workshop on Statistical Machine Translation, pages 10–51, Montr´eal, Canada, June 7-8, 2012. c⃝2012 Association for Computational Linguistics Findings of the 2012 Workshop on Statistical Machine Translation Chris Callison-Burch Johns Hopkins University Philipp Koehn University of Edinburgh Christof Monz University of Amsterdam Matt Post Johns Hopkins University Radu Soricut SDL Language Weaver Lucia Specia University of Sheffield Abstract This paper presents the results of the WMT12 shared tasks, which included a translation task, a task for machine translation evaluation metrics, and a task for run-time estimation of machine translation quality. We conducted a large-scale manual evaluation of 103 machine translation systems submitted by 34 teams. We used the ranking of these systems to mea- sure how strongly automatic metrics correlate with human judgments of translation quality for 12 evaluation metrics. We introduced a new quality estimation task this year, and eval- uated submissions from 11 teams. 1 Introduction This paper presents the results of the shared tasks of the Workshop on statistical Machine Translation (WMT), which was held at NAACL 2012. This workshop builds on six previous WMT workshops (Koehn and Monz, 2006; Callison-Burch et al., 2007; Callison-Burch et al., 2008; Callison-Burch et al., 2009; Callison-Burch et al., 2010; Callison- Burch et al., 2011). In the past, the workshops have featured a number of shared tasks: a translation task between English and other languages, a task for au- tomatic evaluation metrics to predict human judg- ments of translation quality, and a system combina- tion task to get better translation quality by combin- ing the outputs of multiple translation systems. This year we discontinued the system combination task, and introduced a new task in its place: • Quality estimation task – Structured predic- tion tasks like MT are difficult, but the dif- ficulty is not uniform across all input types. It would thus be useful to have some mea- sure of confidence in the quality of the output, which has potential usefulness in a range of set- tings, such as deciding whether output needs human post-editing or selecting the best trans- lation from outputs from a number of systems. This shared task focused on sentence-level es- timation, and challenged participants to rate the quality of sentences produced by a stan- dard Moses translation system on an English- Spanish news corpus in one of two tasks: ranking and scoring. Predictions were scored against a blind test set manually annotated with relevant quality judgments. The primary objectives of WMT are to evaluate the state of the art in machine translation, to dissem- inate common test sets and public training data with published performance numbers, and to refine eval- uation methodologies for machine translation. As with previous workshops, all of the data, transla- tions, and collected human judgments are publicly available.1 We hope these datasets form a valuable resource for research into statistical machine transla- tion, system combination, and automatic evaluation or automatic prediction of translation quality. 2 Overview of the Shared Translation Task The recurring task of the workshop examines …

Content truncated to 500 words