Results of the WMT13 Metrics Shared Task

Results of the WMT13 Metrics Shared Task Matouˇs Mach´aˇcek and Ondˇrej Bojar Charles University in Prague, Faculty of Mathematics and Physics Institute of Formal and Applied Linguistics machacekmatous@gmail.com and bojar@ufal.mff.cuni.cz Abstract This paper presents the results of the WMT13 Metrics Shared Task. We asked participants of this task to score the outputs of the MT systems involved in WMT13 Shared Translation Task. We collected scores of 16 metrics from 8 re- search groups. In addition to that we com- puted scores of 5 standard metrics such as BLEU, WER, PER as baselines. Collected scores were evaluated in terms of system level correlation (how well each metric’s scores correlate with WMT13 official hu- man scores) and in terms of segment level correlation (how often a metric agrees with humans in comparing two translations of a particular sentence). This is a corrected version of January 20, 2014. 1 Introduction Automatic machine translation metrics play a very important role in the development of MT systems and their evaluation. There are many different metrics of diverse nature and one would like to assess their quality. For this reason, the Metrics Shared Task is held annually at the Workshop of Statistical Machine Translation (Callison-Burch et al., 2012). This year, the Metrics Task was run by different organizers but the only visible change is hopefully that the results of the task are pre- sented in a separate paper instead of the main WMT overview paper. In this task, we asked metrics developers to score the outputs of WMT13 Shared Translation Task (Bojar et al., 2013). We have collected the computed metrics’ scores and use them to evalu- ate quality of the metrics. The systems’ outputs, human judgements and evaluated metrics are described in Section 2. The quality of the metrics in terms of system level cor- relation is reported in Section 3. Segment level correlation is reported in Section 4. 2 Data We used the translations of MT systems involved in WMT13 Shared Translation Task together with reference translations as the test set for the Metrics Task. This dataset consists of 135 systems’ out- puts and 6 reference translations in 10 translation directions (5 into English and 5 out of English). Each system’s output and the reference translation contain 3000 sentences. For more details please see the WMT13 main overview paper (Bojar et al., 2013). 2.1 Manual MT Quality Judgements During the WMT13 Translation Task a large scale manual annotation was conducted to compare the systems. We used these collected human judge- ments for evaluating the automatic metrics. The participants in the manual annotation were asked to evaluate system outputs by ranking trans- lated sentences relative to each other. For each source segment that was included in the procedure, the annotator was shown the outputs of five sys- tems to which he or she was supposed to assign ranks. Ties were allowed. Only sentences with 30 or less words were ranked by humans. These collected rank labels were then used to assign each system a score …

Content truncated to 500 words