Findings of the 2017 Conference on Machine Translation (WMT17)

Proceedings of the Conference on Machine Translation (WMT), Volume 2: Shared Task Papers, pages 169–214 Copenhagen, Denmark, September 711, 2017. c⃝2017 Association for Computational Linguistics Findings of the 2017 Conference on Machine Translation (WMT17) Ondˇrej Bojar Charles University Rajen Chatterjee FBK Christian Federmann Microsoft Research Yvette Graham Dublin City University Barry Haddow Univ. of Edinburgh Shujian Huang Nanjing University Matthias Huck LMU Munich Philipp Koehn JHU / Edinburgh Qun Liu Dublin City University Varvara Logacheva MIPT Moscow Christof Monz Univ. of Amsterdam Matteo Negri FBK Matt Post Johns Hopkins Univ. Raphael Rubino DFKI & Saarland Univ. Lucia Specia Univ. of Sheffield Marco Turchi FBK Abstract This paper presents the results of the WMT17 shared tasks, which included three machine translation (MT) tasks (news, biomedical, and multimodal), two evaluation tasks (metrics and run-time es- timation of MT quality), an automatic post-editing task, a neural MT training task, and a bandit learning task. 1 Introduction We present the results of the shared tasks of the Second Conference on Statistical Machine Trans- lation (WMT) held at EMNLP 2017. This confer- ence builds on eleven previous editions of WMT as workshops and conference (Koehn and Monz, 2006; Callison-Burch et al., 2007, 2008, 2009, 2010, 2011, 2012; Bojar et al., 2013, 2014, 2015, 2016a). This year we conducted several official tasks. We report in this paper on three tasks: • news translation (Section 2, Section 3) • quality estimation (Section 4) • automatic post-editing (Section 5) The conference featured additional shared tasks that are described in separate papers in these pro- ceedings: • metrics (Bojar et al., 2017a) • multimodal machine translation and multilin- gual image description (Elliott et al., 2017) • biomedical translation (Jimeno Yepes et al., 2017) • neural MT training (Bojar et al., 2017b) • bandit learning (Sokolov et al., 2017) In the news translation task (Section 2), partic- ipants were asked to translate a shared test set, optionally restricting themselves to the provided training data (constraint condition). We held 14 translation tasks this year, between English and each of Chinese, Czech, German, Finnish, Lat- vian, Russian, and Turkish. The Latvian and Chi- nese translation tasks were new this year. Latvian is a lesser resourced data condition on challenging language pair. Chinese allowed us to co-operate with an ongoing evaluation campaign on Asian languages organized alongside the Chinese Work- shop on Machine Translation (CWMT).1 System outputs for each task were evaluated both automat- ically and manually. The human evaluation (Section 3) involves ask- ing human judges to score sentences output by anonymized systems. We obtained large numbers of assessments from researchers who contributed evaluations proportional to the number of tasks they entered. In addition, we used Mechanical Turk to collect further evaluations. This year, the official manual evaluation metric is based on judg- ments of adequacy on a 100-point scale, a method we explored last year with convincing results in terms of the trade-off between annotation effort and reliable distinctions between systems. The quality estimation task (Section 4) this year included three …

Content truncated to 500 words