Findings of the 2011 Workshop on Statistical Machine Translation

Proceedings of the 6th Workshop on Statistical Machine Translation, pages 22–64, Edinburgh, Scotland, UK, July 30–31, 2011. c⃝2011 Association for Computational Linguistics Findings of the 2011 Workshop on Statistical Machine Translation Chris Callison-Burch Center for Language and Speech Processing Johns Hopkins University Philipp Koehn School of Informatics University of Edinburgh Christof Monz Informatics Institute University of Amsterdam Omar F. Zaidan Center for Language and Speech Processing Johns Hopkins University Abstract This paper presents the results of the WMT11 shared tasks, which included a translation task, a system combination task, and a task for machine translation evaluation metrics. We conducted a large-scale manual evaluation of 148 machine translation systems and 41 sys- tem combination entries. We used the rank- ing of these systems to measure how strongly automatic metrics correlate with human judg- ments of translation quality for 21 evaluation metrics. This year featured a Haitian Creole to English task translating SMS messages sent to an emergency response service in the af- termath of the Haitian earthquake. We also conducted a pilot ‘tunable metrics’ task to test whether optimizing a fixed system to differ- ent metrics would result in perceptibly differ- ent translation quality. 1 Introduction This paper presents the results of the shared tasks of the Workshop on statistical Machine Translation (WMT), which was held at EMNLP 2011. This workshop builds on five previous WMT workshops (Koehn and Monz, 2006; Callison-Burch et al., 2007; Callison-Burch et al., 2008; Callison-Burch et al., 2009; Callison-Burch et al., 2010). The work- shops feature three shared tasks: a translation task between English and other languages, a task to com- bine the output of multiple machine translation sys- tems, and a task to predict human judgments of translation quality using automatic evaluation met- rics. The performance for each of these shared tasks is determined through a comprehensive human eval- uation. There were a two additions to this year’s workshop that were not part of previous workshops: • Haitian Creole featured task – In addition to translation between European language pairs, we featured a new translation task: translating Haitian Creole SMS messages that were sent to an emergency response hotline in the im- mediate aftermath of the 2010 Haitian earth- quake. The goal of this task is to encourage re- searchers to focus on challenges that may arise in future humanitarian crises. We invited Will Lewis, Rob Munro and Stephan Vogel to pub- lish a paper about their experience developing translation technology in response to the crisis (Lewis et al., 2011). They provided the data used in the Haitian Creole featured translation task. We hope that the introduction of this new dataset will provide a testbed for dealing with low resource languages and the informal lan- guage usage found in SMS messages. • Tunable metric shared task – We conducted a pilot of a new shared task to use evaluation metrics to tune the parameters of a machine translation system. Although previous work- shops have shown evaluation metrics other than BLEU are more strongly correlated with …

Content truncated to 500 words