Findings of the 2013 Workshop on Statistical Machine Translation

Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 1–44, Sofia, Bulgaria, August 8-9, 2013 c⃝2013 Association for Computational Linguistics Findings of the 2013 Workshop on Statistical Machine Translation Ondˇrej Bojar Charles University in Prague Christian Buck University of Edinburgh Chris Callison-Burch University of Pennsylvania Christian Federmann Saarland University Barry Haddow University of Edinburgh Philipp Koehn University of Edinburgh Christof Monz University of Amsterdam Matt Post Johns Hopkins University Radu Soricut Google Lucia Specia University of Sheffield Abstract We present the results of the WMT13 shared tasks, which included a translation task, a task for run-time estimation of ma- chine translation quality, and an unoffi- cial metrics task. This year, 143 machine translation systems were submitted to the ten translation tasks from 23 institutions. An additional 6 anonymized systems were included, and were then evaluated both au- tomatically and manually, in our largest manual evaluation to date. The quality es- timation task had four subtasks, with a to- tal of 14 teams, submitting 55 entries. 1 Introduction We present the results of the shared tasks of the Workshop on Statistical Machine Translation (WMT) held at ACL 2013. This workshop builds on seven previous WMT workshops (Koehn and Monz, 2006; Callison-Burch et al., 2007, 2008, 2009, 2010, 2011, 2012). This year we conducted three official tasks: a translation task, a human evaluation of transla- tion results, and a quality estimation task.1 In the translation task (§2), participants were asked to translate a shared test set, optionally restrict- ing themselves to the provided training data. We held ten translation tasks this year, between En- glish and each of Czech, French, German, Span- ish, and Russian. The Russian translation tasks were new this year, and were also the most popu- lar. The system outputs for each task were evalu- ated both automatically and manually. The human evaluation task (§3) involves ask- ing human judges to rank sentences output by anonymized systems. We obtained large numbers of rankings from two groups: researchers (who 1The traditional metrics task is evaluated in a separate pa- per (Mach´aˇcek and Bojar, 2013). contributed evaluations proportional to the number of tasks they entered) and workers on Amazon’s Mechanical Turk (who were paid). This year’s ef- fort was our largest yet by a wide margin; we man- aged to collect an order of magnitude more judg- ments than in the past, allowing us to achieve sta- tistical significance on the majority of the pairwise system rankings. This year, we are also clustering the systems according to these significance results, instead of presenting a total ordering over systems. The focus of the quality estimation task (§6) is to produce real-time estimates of sentence- or word-level machine translation quality. This task has potential usefulness in a range of settings, such as prioritizing output for human post-editing, or selecting the best translations from a number of systems. This year the following subtasks were proposed: prediction of percentage of word edits necessary to fix a sentence, ranking of up to five al- ternative …

Content truncated to 500 words