Results of the WMT15 Metrics Shared Task

Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 256–273, Lisboa, Portugal, 17-18 September 2015. c⃝2015 Association for Computational Linguistics. Results of the WMT15 Metrics Shared Task Miloˇs Stanojevi´c and Amir Kamran University of Amsterdam ILLC {m.stanojevic,a.kamran}@uva.nl Philipp Koehn Johns Hopkins University DCS phi@jhu.edu Ondˇrej Bojar Charles University in Prague MFF ´UFAL bojar@ufal.mff.cuni.cz Abstract This paper presents the results of the WMT15 Metrics Shared Task. We asked participants of this task to score the out- puts of the MT systems involved in the WMT15 Shared Translation Task. We col- lected scores of 46 metrics from 11 re- search groups. In addition to that, we computed scores of 7 standard metrics (BLEU, SentBLEU, NIST, WER, PER, TER and CDER) as baselines. The col- lected scores were evaluated in terms of system level correlation (how well each metric’s scores correlate with WMT15 of- ficial manual ranking of systems) and in terms of segment level correlation (how often a metric agrees with humans in com- paring two translations of a particular sen- tence). 1 Introduction Automatic machine translation metrics play a very important role in the development of MT systems and their evaluation. There are many different metrics of diverse nature and one would like to assess their quality. For this reason, the Met- rics Shared Task is held annually at the Work- shop of Statistical Machine Translation1, starting with Koehn and Monz (2006) and following up to Mach´aˇcek and Bojar (2014). The systems’ outputs, human judgements and evaluated metrics are described in Section 2. The quality of the metrics in terms of system level cor- relation is reported in Section 3. Section 4 is de- voted to segment level correlation. 2 Data We used the translations of MT systems involved in WMT15 Shared Translation Task (Bojar et al., 1http://www.statmt.org/wmt15 2015) together with reference translations as the test set for the Metrics Task. This dataset con- sists of 87 systems’ outputs and 10 reference trans- lations in 10 translation directions (English from and into Czech, Finnish, French, German and Rus- sian). The number of sentences in system and ref- erence translations varies among language pairs ranging from 1370 for Finnish-English to 2818 for Russian-English. For more details, please see the WMT15 overview paper (Bojar et al., 2015). 2.1 Manual MT Quality Judgements During the WMT15 Translation Task, a large scale manual annotation was conducted to compare the translation quality of participating systems. We used these collected human judgements for the evaluation of the automatic metrics. The participants in the manual annotation were asked to evaluate system outputs by ranking trans- lated sentences relative to each other. For each source segment that was included in the proce- dure, the annotator was shown five different out- puts to which he or she was supposed to assign ranks. Ties were allowed. These collected rank labels for each five-tuple of outputs were then interpreted as pairwise com- parisons of systems and used to assign each sys- tem a score that reflects how high that system was …

Content truncated to 500 words