Results of the WMT14 Metrics Shared Task

Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 293–301, Baltimore, Maryland USA, June 26–27, 2014. c⃝2014 Association for Computational Linguistics Results of the WMT14 Metrics Shared Task Matouˇs Mach´aˇcek and Ondˇrej Bojar Charles University in Prague, Faculty of Mathematics and Physics Institute of Formal and Applied Linguistics machacekmatous@gmail.com and bojar@ufal.mff.cuni.cz Abstract This paper presents the results of the WMT14 Metrics Shared Task. We asked participants of this task to score the outputs of the MT systems involved in WMT14 Shared Translation Task. We col- lected scores of 23 metrics from 12 re- search groups. In addition to that we com- puted scores of 6 standard metrics (BLEU, NIST, WER, PER, TER and CDER) as baselines. The collected scores were eval- uated in terms of system level correlation (how well each metric’s scores correlate with WMT14 official manual ranking of systems) and in terms of segment level correlation (how often a metric agrees with humans in comparing two translations of a particular sentence). 1 Introduction Automatic machine translation metrics play a very important role in the development of MT systems and their evaluation. There are many different metrics of diverse nature and one would like to assess their quality. For this reason, the Met- rics Shared Task is held annually at the Workshop of Statistical Machine Translation1, starting with Koehn and Monz (2006) and following up to Bo- jar et al. (2014). In this task, we asked metrics developers to score the outputs of WMT14 Shared Translation Task (Bojar et al., 2014). We have collected the computed metrics’ scores and use them to evalu- ate quality of the metrics. The systems’ outputs, human judgements and evaluated metrics are described in Section 2. The quality of the metrics in terms of system level cor- relation is reported in Section 3. Segment level correlation with a detailed discussion and a slight 1http://www.statmt.org/wmt13 change in the calculation compared to the previous year is reported in Section 4. 2 Data We used the translations of MT systems involved in WMT14 Shared Translation Task together with reference translations as the test set for the Met- rics Task. This dataset consists of 110 systems’ outputs and 10 reference translations in 10 trans- lation directions (English from and into Czech, French, German, Hindi and Russian). For most of the translation directions each system’s output and the reference translation contain 3003 sentences. For more details please see the WMT14 overview paper (Bojar et al., 2014). 2.1 Manual MT Quality Judgements During the WMT14 Translation Task, a large scale manual annotation was conducted to compare the systems. We used these collected human judge- ments for the evalution of the automatic metrics. The participants in the manual annotation were asked to evaluate system outputs by ranking trans- lated sentences relative to each other. For each source segment that was included in the procedure, the annotator was shown the outputs of five sys- tems to which he or she was supposed to assign ranks. Ties were allowed. These collected rank …

Content truncated to 500 words