Putting Human Assessments of Machine Translation Systems in Order

Proceedings of the 7th Workshop on Statistical Machine Translation, pages 1–9, Montr´eal, Canada, June 7-8, 2012. c⃝2012 Association for Computational Linguistics Putting Human Assessments of Machine Translation Systems in Order Adam Lopez Human Language Technology Center of Excellence Johns Hopkins University Abstract Human assessment is often considered the gold standard in evaluation of translation sys- tems. But in order for the evaluation to be meaningful, the rankings obtained from human assessment must be consistent and repeatable. Recent analysis by Bojar et al. (2011) raised several concerns about the rankings derived from human assessments of English-Czech translation systems in the 2010 Workshop on Machine Translation. We extend their analysis to all of the ranking tasks from 2010 and 2011, and show through an exten- sion of their reasoning that the ranking is nat- urally cast as an instance of finding the mini- mum feedback arc set in a tournament, a well- known NP-complete problem. All instances of this problem in the workshop data are ef- ficiently solvable, but in some cases the rank- ings it produces are surprisingly different from the ones previously published. This leads to strong caveats and recommendations for both producers and consumers of these rankings. 1 Introduction The value of machine translation depends on its util- ity to human users, either directly through their use of it, or indirectly through downstream tasks such as cross-lingual information extraction or retrieval. It is therefore essential to assess machine transla- tion systems according to this utility, but there is a widespread perception that direct human assessment is costly, unreproducible, and difficult to interpret. Automatic metrics that predict human utility have therefore attracted substantial attention since they are at least cheap and reproducible given identical data conditions, though they are frequently and cor- rectly criticized for low interpretability and correla- tion with true utility. Their use (and abuse) remains contentious. The organizers of the annual Workshop on Ma- chine Translation (WMT) have taken a strong stance in this debate, asserting the primacy of human eval- uation. Every annual report of their findings since 2007 has included a variant of the following state- ment: It is our contention that automatic mea- sures are an imperfect substitute for hu- man assessment of translation quality. Therefore, we define the manual evalua- tion to be primary, and use the human judgments to validate automatic metrics. (Callison-Burch et al., 2011) The workshop’s human evaluation component has been gradually refined over several years, and as a consequence it has produced a fantastic collection of publicly available data consisting primarily of pair- wise judgements of translation systems made by hu- man assessors across a wide variety of languages and tasks. Despite superb effort in the collection of these assessments, less attention has been focused on the final product derived from them: a totally- ordered ranking of translation systems participating in each task. Many of the official workshop results depend crucially on this ranking, including the eval- uation of both machine translation systems and auto- matic metrics. Considering the …

Content truncated to 500 words