Findings of the 2015 Workshop on Statistical Machine Translation

Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 1–46, Lisboa, Portugal, 17-18 September 2015. c⃝2015 Association for Computational Linguistics. Findings of the 2015 Workshop on Statistical Machine Translation Ondˇrej Bojar Charles Univ. in Prague Rajen Chatterjee FBK Christian Federmann Microsoft Research Barry Haddow Univ. of Edinburgh Matthias Huck Univ. of Edinburgh Chris Hokamp Dublin City Univ. Philipp Koehn JHU / Edinburgh Varvara Logacheva Univ. of Sheffield Christof Monz Univ. of Amsterdam Matteo Negri FBK Matt Post Johns Hopkins Univ. Carolina Scarton Univ. of Sheffield Lucia Specia Univ. of Sheffield Marco Turchi FBK Abstract This paper presents the results of the WMT15 shared tasks, which included a standard news translation task, a metrics task, a tuning task, a task for run-time estimation of machine translation quality, and an automatic post-editing task. This year, 68 machine translation systems from 24 institutions were submitted to the ten translation directions in the standard trans- lation task. An additional 7 anonymized systems were included, and were then evaluated both automatically and manu- ally. The quality estimation task had three subtasks, with a total of 10 teams, submit- ting 34 entries. The pilot automatic post- editing task had a total of 4 teams, submit- ting 7 entries. 1 Introduction We present the results of the shared tasks of the Workshop on Statistical Machine Translation (WMT) held at EMNLP 2015. This workshop builds on eight previous WMT workshops (Koehn and Monz, 2006; Callison-Burch et al., 2007, 2008, 2009, 2010, 2011, 2012; Bojar et al., 2013, 2014). This year we conducted five official tasks: a translation task, a quality estimation task, a met- rics task, a tuning task1, and a automatic post- editing task. In the translation task (§2), participants were asked to translate a shared test set, optionally re- stricting themselves to the provided training data. We held ten translation tasks this year, between English and each of Czech, French, German, Finnish, and Russian. The Finnish translation 1The metrics and tuning tasks are reported in separate pa- pers (Stanojevi´c et al., 2015a,b). tasks were new this year, providing a lesser re- sourced data condition on a challenging language pair. The system outputs for each task were evalu- ated both automatically and manually. The human evaluation (§3) involves asking human judges to rank sentences output by anonymized systems. We obtained large num- bers of rankings from researchers who contributed evaluations proportional to the number of tasks they entered. We made data collection more ef- ficient and used TrueSkill as ranking method. The quality estimation task (§4) this year in- cluded three subtasks: sentence-level prediction of post-editing effort scores, word-level prediction of good/bad labels, and document-level prediction of Meteor scores. Datasets were released with English→Spanish news translations for sentence and word level, English↔German news transla- tions for document level. The first round of the automatic post-editing task (§5) examined automatic methods for cor- recting errors produced by an unknown machine translation system. Participants were provided with training triples containing source, target and human post-editions, and were asked …

Content truncated to 500 words