Results of the WMT17 Metrics Shared Task

Proceedings of the Conference on Machine Translation (WMT), Volume 2: Shared Task Papers, pages 489–513 Copenhagen, Denmark, September 711, 2017. c⃝2017 Association for Computational Linguistics Results of the WMT17 Metrics Shared Task Ondˇrej Bojar Charles University MFF ´UFAL bojar@ufal.mff.cuni.cz Yvette Graham Dublin City University ADAPT graham.yvette@gmail.com Amir Kamran University of Amsterdam ILLC a.kamran@uva.nl Abstract This paper presents the results of the WMT17 Metrics Shared Task. We asked participants of this task to score the out- puts of the MT systems involved in the WMT17 news translation task and Neu- ral MT training task. We collected scores of 14 metrics from 8 research groups. In addition to that, we computed scores of 7 standard metrics (BLEU, SentBLEU, NIST, WER, PER, TER and CDER) as baselines. The collected scores were eval- uated in terms of system-level correlation (how well each metric’s scores correlate with WMT17 official manual ranking of systems) and in terms of segment level correlation (how often a metric agrees with humans in judging the quality of a partic- ular sentence). This year, we build upon two types of manual judgements: direct assessment (DA) and HUME manual semantic judge- ments. 1 Introduction Evaluating the quality of machine translation (MT) is critical for developers of MT systems to monitor progress as well as for MT users to select among available MT engines for their language pair of interest. Manual evaluation is however costly and difficult to reproduce. Automatic MT evaluation can resolve these issues, if it matches manual evaluation. The Metrics Shared Task1 of WMT annually evaluates the performance of au- tomatic machine translation metrics in their abil- ity to provide a substitute for human assessment of translation quality. 1http://www.statmt.org/wmt17/ metrics-task.html, starting with Koehn and Monz (2006) up to Bojar et al. (2016b) In contrast to MT quality estimation, the metrics task provides participating metrics with reference translations with which MT outputs are compared. The metrics task itself then needs manual judge- ments of translation quality in order to check the extent to which the automatic metrics can approx- imate the judgement. For situations where the ref- erence translation is not available, please consult the results of Quality Estimation Task (Bojar et al., 2017a). We keep the two main types of metric eval- uation unchanged from the previous years. In system-level evaluation, each metric provides a quality score for the whole translated test set (usu- ally a set of documents, in fact). In segment-level evaluation, a score has to be assigned to every in- dividual sentence. The underlying texts and MT systems come from two other WMT tasks, namely News Trans- lation Task (Bojar et al., 2017a, denoted as Find- ings 2017 in the following) and Neural MT train- ing task (Bojar et al., 2017b), and from the EU project HimL, aiming at translation of health- related documents. The texts were drawn mainly from the news domain and, to a limited extent, from the medical domain and involve translations to/from Chinese (zh), Czech (cs), Finnish (fi), Ger- man (de), Latvian (lv), …

Content truncated to 500 words