Crowd-Sourcing of Human Judgments of Machine Translation Fluency

Crowd-Sourcing of Human Judgments of Machine Translation Fluency Yvette Graham Timothy Baldwin Alistair Moffat Justin Zobel Department of Computing and Information Systems, The University of Melbourne {ygraham,tbaldwin,ammoffat,jzobel}@unimelb.edu.au Abstract Human evaluation of machine translation quality is a key element in the develop- ment of machine translation systems, as automatic metrics are validated through correlation with human judgment. How- ever, achievement of consistent human judgments of machine translation is not easy, with decreasing levels of consis- tency reported in annual evaluation cam- paigns. In this paper we describe experi- ences gained during the collection of hu- man judgments of the fluency of machine translation output using Amazon’s Me- chanical Turk service. We gathered a large collection of crowd-sourced human judg- ments for the machine translation systems that participated in the WMT 2012 shared translation task, collected across a range of eight different assessment configurations to gain insight into possible causes of – and remedies for – inconsistency in human judgments. Overall, approximately half of the workers carry out the human evalua- tion to a high standard, but effectiveness varies considerably across different target languages, with dramatically higher num- bers of good quality judgments for Span- ish and French, and the reverse observed for German. 1 Introduction The ability to accurately measure the properties of an object of study, such as a computational sys- tem, is fundamental to progress in science. For measurements to be meaningful, they need to be comparable between systems, and to be an accu- rate proxy for the properties of the systems being studied. For machine translation (MT), measurement has been a combination of human judgments and automated measurements. With the aim of re- moving system biases and creating robust compar- isons, there has been extensive use of workshops and shared tasks such as the ongoing Workshops on Statistical Machine Translation (WMT) and the NIST Open Machine Translation (OpenMT) eval- uations. The basis of system evaluation is gener- ally human judgments, which have also been used to evaluate automatic metrics such as BLEU (Pa- pineni et al., 2001), under the assumption that a metric that correlates strongly with human judg- ments is more valid than a metric with weak cor- relation. Human evaluation of MT thus forms the foundation of evaluation in empirical MT, regard- less of whether a particular evaluation makes use of human judges or automatic metrics. The current methodology used for the task of human evaluation in MT is problematic, however, as assessments carried out by expert judges are highly inconsistent. Even when a single expert judge is asked to assess the same pair of trans- lations in two separate sittings, the second judg- ment is often at odds with the initial one (Bojar et al., 2013). Somewhat paradoxically, and despite the fact that experts are not consistent, when non- experts are employed to do judgments, there is a tendency to give preference to non-experts who demonstrate high agreement with experts. We have used Amazon’s Mechanical Turk ser- vice (AMT) to gather human judgments of ma- …

Content truncated to 500 words