A Grain of Salt for the WMT Manual Evaluation

Proceedings of the 6th Workshop on Statistical Machine Translation, pages 1–11, Edinburgh, Scotland, UK, July 30–31, 2011. c⃝2011 Association for Computational Linguistics A Grain of Salt for the WMT Manual Evaluation∗ Ondˇrej Bojar, Miloˇs Ercegovˇcevi´c, Martin Popel Charles University in Prague Faculty of Mathematics and Physics Institute of Formal and Applied Linguistics {bojar,popel}@ufal.mff.cuni.cz ercegovcevic@hotmail.com Omar F. Zaidan Department of Computer Science Johns Hopkins University ozaidan@cs.jhu.edu Abstract The Workshop on Statistical Machine Translation (WMT) has become one of ACL’s flagship workshops, held annually since 2006. In addition to soliciting pa- pers from the research community, WMT also features a shared translation task for evaluating MT systems. This shared task is notable for having manual evaluation as its cornerstone. The Workshop’s overview paper, playing a descriptive and adminis- trative role, reports the main results of the evaluation without delving deep into ana- lyzing those results. The aim of this paper is to investigate and explain some interest- ing idiosyncrasies in the reported results, which only become apparent when per- forming a more thorough analysis of the collected annotations. Our analysis sheds some light on how the reported results should (and should not) be interpreted, and also gives rise to some helpful recommen- dation for the organizers of WMT. 1 Introduction The Workshop on Statistical Machine Translation (WMT) has become an annual feast for MT re- searchers. Of particular interest is WMT’s shared translation task, featuring a component for man- ual evaluation of MT systems. The friendly com- petition is a source of inspiration for participating teams, and the yearly overview paper (Callison- Burch et al., 2010) provides a concise report of the state of the art. However, the amount of interest- ing data collected every year (the system outputs ∗ This work has been supported by the grants EuroMa- trixPlus (FP7-ICT-2007-3-231720 of the EU and 7E09003 of the Czech Republic), P406/10/P259, MSM 0021620838, and DARPA GALE program under Contract No. HR0011-06-2- 0001. We are grateful to our students, colleagues, and the three reviewers for various observations and suggestions. and, most importantly, the annotator judgments) is quite large, exceeding what the WMT overview paper can afford to analyze with much depth. In this paper, we take a closer look at the data collected in last year’s workshop, WMT101, and delve a bit deeper into analyzing the manual judg- ments. We focus mainly on the English-to-Czech task, as it included a diverse portfolio of MT sys- tems, was a heavily judged language pair, and also illustrates interesting “contradictions” in the re- sults. We try to explain such points of interest, and analyze what we believe to be the positive and negative aspects of the currently established eval- uation procedure of WMT. Section 2 examines the primary style of man- ual evaluation: system ranking. We discuss how the interpretation of collected judgments, the com- putation of annotator agreement, and document that annotators’ individual preferences may render two systems effectively incomparable. Section 3 is devoted to the impact of embedding reference translations, while Section 4 and …

Content truncated to 500 words