Efficient Elicitation of Annotations for Human Evaluation of Machine Translation

Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 1–11, Baltimore, Maryland USA, June 26–27, 2014. c⃝2014 Association for Computational Linguistics Efficient Elicitation of Annotations for Human Evaluation of Machine Translation Keisuke Sakaguchi∗, Matt Post†, Benjamin Van Durme† ∗ Center for Language and Speech Processing † Human Language Technology Center of Excellence Johns Hopkins University, Baltimore, Maryland {keisuke,post,vandurme}@cs.jhu.edu Abstract A main output of the annual Workshop on Statistical Machine Translation (WMT) is a ranking of the systems that partici- pated in its shared translation tasks, pro- duced by aggregating pairwise sentence- level comparisons collected from human judges. Over the past few years, there have been a number of tweaks to the ag- gregation formula in attempts to address issues arising from the inherent ambigu- ity and subjectivity of the task, as well as weaknesses in the proposed models and the manner of model selection. We continue this line of work by adapt- ing the TrueSkillTM algorithm — an online approach for modeling the relative skills of players in ongoing competitions, such as Microsoft’s Xbox Live — to the hu- man evaluation of machine translation out- put. Our experimental results show that TrueSkill outperforms other recently pro- posed models on accuracy, and also can significantly reduce the number of pair- wise annotations that need to be collected by sampling non-uniformly from the space of system competitions. 1 Introduction The Workshop on Statistical Machine Translation (WMT) has long been a central event in the ma- chine translation (MT) community for the evalua- tion of MT output. It hosts an annual set of shared translation tasks focused mostly on the translation of western European languages. One of its main functions is to publish a ranking of the systems for each task, which are produced by aggregating a large number of human judgments of sentence- level pairwise rankings of system outputs. While the performance on many automatic metrics is also # score range system 1 0.638 1 UEDIN-HEAFIELD 2 0.604 2-3 UEDIN 0.591 2-3 ONLINE-B 4 0.571 4-5 LIMSI-SOUL 0.562 4-5 KIT 0.541 5-6 ONLINE-A 7 0.512 7 MES-SIMPLIFIED 8 0.486 8 DCU 9 0.439 9-10 RWTH 0.429 9-11 CMU-T2T 0.420 10-11 CU-ZEMAN 12 0.389 12 JHU 13 0.322 13 SHEF-WPROA Table 1: System rankings presented as clusters (WMT13 French-English competition). The score column is the percentage of time each system was judged better across its comparisons (§2.1). reported (e.g., BLEU (Papineni et al., 2002)), the human evaluation is considered primary, and is in fact used as the gold standard for its metrics task, where evaluation metrics are evaluated. In machine translation, the longstanding dis- agreements about evaluation measures do not go away when moving from automatic metrics to hu- man judges. This is due in no small part to the in- herent ambiguity and subjectivity of the task, but also arises from the particular way that the WMT organizers produce the rankings. The system- level rankings are produced by collecting pairwise sentence-level comparisons between system out- puts. These are then aggregated to …

Content truncated to 500 words