Re-assessing the WMT2013 Human Evaluation with Professional Translators Trainees

Re-assessing the WMT2013 Human Evaluation with Professional Translators Trainees Mihaela Vela Saarland University m.vela@mx.uni-saarland.de Josef van Genabith German Research Center for Artificial Intelligence Josef.Van_Genabith@dfki.de Abstract This paper presents experiments on the human ranking task performed during WMT2013. The goal of these experiments is to re-run the human evaluation task with translation studies students and to compare the results with the human rankings per- formed by the WMT development teams during WMT2013. More specifically, we test whether we can reproduce, and if yes to what extent, the WMT2013 ranking task and whether specialised knowledge from translation studies influences the re- sults in terms of intra- and inter-annotator agreement as well as in terms of system ranking. We present two experiments on the English-German WMT2013 machine translation output. Analysis of the data follows the methods described in the of- ficial WMT2013 report. The results in- dicate a higher inter- and intra-annotator agreement, less ties and slight differences in ranking for the translation studies stu- dents as compared to the WMT develop- ment teams. 1 Introduction Machine translation evaluation is an important el- ement in the process of building MT systems. The Workshop for Statistical Machine Translation (WMT) compares new techniques for MT through human and automatic MT evaluation and provides also tracks for evaluation metrics, quality estima- tion of MT as well as post-editing of MT. To date, the most popular MT evaluation met- rics essentially measure lexical overlap between reference and hypothesis translation such as IBM c⃝ 2015 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC- BY-ND. BLEU (Papineni et al., 2002), NIST (Dodding- ton, 2002), Meteor (Denkowski and Lavie, 2014), WER (Levenshtein, 1966), position-independent error rate metric PER (Tillmann et al., 1997) and the translation edit rate metric TER (Snover et al., 2006) and TERp (Snover et al., 2009). Gonzàlez et al. (2014) as well as Comelles and Atserias (2014) introduce their fully automatic approaches to ma- chine translation evaluation using lexical, syntac- tic and semantic information when comparing the machine translation output with reference transla- tions. Human machine translation evaluation can be performed with different methods. Lo and Wu (2011) propose HMEANT, a metric based on MEANT (Lo et al., 2012) that measures mean- ing preservation between hypothesis and reference translation on the basis of verb frames and their role fillers. Another method is HTER (Snover et al., 2006) which produces targeted reference translations by post-editing MT output. Another method is HTER (Snover et al., 2006) which produces targeted reference translations by post- editing MT output. Human evaluation can also be performed by measuring post-editing time, or by asking evaluators to assess the fluency and ade- quacy of a hypothesis translation on a Likert scale. Another popular human evaluation method is rank- ing: ordering a set of translation hypotheses ac- cording to their quality. This is also the method ap- plied during the recent WMTs, where humans are asked to rank machine translation output by using APPRAISE (Federmann, 2012), a …

Content truncated to 500 words