Findings of the 2014 Workshop on Statistical Machine Translation

Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 12–58, Baltimore, Maryland USA, June 26–27, 2014. c⃝2014 Association for Computational Linguistics Findings of the 2014 Workshop on Statistical Machine Translation Ondˇrej Bojar Charles University in Prague Christian Buck University of Edinburgh Christian Federmann Microsoft Research Barry Haddow University of Edinburgh Philipp Koehn JHU / Edinburgh Johannes Leveling Dublin City University Christof Monz University of Amsterdam Pavel Pecina Charles University in Prague Matt Post Johns Hopkins University Herve Saint-Amand University of Edinburgh Radu Soricut Google Lucia Specia University of Sheffield Aleˇs Tamchyna Charles University in Prague Abstract This paper presents the results of the WMT14 shared tasks, which included a standard news translation task, a sepa- rate medical translation task, a task for run-time estimation of machine translation quality, and a metrics task. This year, 143 machine translation systems from 23 insti- tutions were submitted to the ten transla- tion directions in the standard translation task. An additional 6 anonymized sys- tems were included, and were then evalu- ated both automatically and manually. The quality estimation task had four subtasks, with a total of 10 teams, submitting 57 en- tries. 1 Introduction We present the results of the shared tasks of the Workshop on Statistical Machine Translation (WMT) held at ACL 2014. This workshop builds on eight previous WMT workshops (Koehn and Monz, 2006; Callison-Burch et al., 2007, 2008, 2009, 2010, 2011, 2012; Bojar et al., 2013). This year we conducted four official tasks: a translation task, a quality estimation task, a met- rics task1 and a medical translation task. In the translation task (§2), participants were asked to translate a shared test set, optionally restricting themselves to the provided training data. We held ten translation tasks this year, between English and each of Czech, French, German, Hindi, and Rus- sian. The Hindi translation tasks were new this year, providing a lesser resourced data condition on a challenging language pair. The system out- puts for each task were evaluated both automati- cally and manually. 1The metrics task is reported in a separate paper (Mach´aˇcek and Bojar, 2014). The human evaluation (§3) involves asking human judges to rank sentences output by anonymized systems. We obtained large num- bers of rankings from researchers who contributed evaluations proportional to the number of tasks they entered. Last year, we dramatically increased the number of judgments, achieving much more meaningful rankings. This year, we developed a new ranking method that allows us to achieve the same with fewer judgments. The quality estimation task (§4) this year included sentence- and word-level subtasks: sentence-level prediction of 1-3 likert scores, sentence-level prediction of percentage of word edits necessary to fix a sentence, sentence-level prediction of post-editing time, and word-level prediction of scores at different levels of granular- ity (correct/incorrect, accuracy/fluency errors, and specific types of errors). Datasets were released with English-Spanish, English-German, Spanish- English and German-English news translations produced by 2-3 machine translation systems and, for some subtasks, a human translation. The medical translation task (§5) was intro- …

Content truncated to 500 words