Fast, Cheap, and Creative: Evaluating Translation Quality Using Amazon’s Mechanical Turk

Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 286–295, Singapore, 6-7 August 2009. c⃝2009 ACL and AFNLP Fast, Cheap, and Creative: Evaluating Translation Quality Using Amazon’s Mechanical Turk Chris Callison-Burch Center for Language and Speech Processing Johns Hopkins University Baltimore, Maryland ccb cs jhu edu Abstract Manual evaluation of translation quality is generally thought to be excessively time consuming and expensive. We explore a fast and inexpensive way of doing it using Amazon’s Mechanical Turk to pay small sums to a large number of non-expert an- notators. For $10 we redundantly recre- ate judgments from a WMT08 transla- tion task. We find that when combined non-expert judgments have a high-level of agreement with the existing gold-standard judgments of machine translation quality, and correlate more strongly with expert judgments than Bleu does. We go on to show that Mechanical Turk can be used to calculate human-mediated translation edit rate (HTER), to conduct reading compre- hension experiments with machine trans- lation, and to create high quality reference translations. 1 Introduction Conventional wisdom holds that manual evalua- tion of machine translation is too time-consuming and expensive to conduct. Instead, researchers routinely use automatic metrics like Bleu (Pap- ineni et al., 2002) as the sole evidence of im- provement to translation quality. Automatic met- rics have been criticized for a variety of reasons (Babych and Hartley, 2004; Callison-Burch et al., 2006; Chiang et al., 2008), and it is clear that they only loosely approximate human judgments. Therefore, having people evaluate translation out- put would be preferable, if it were more practical. In this paper we demonstrate that the manual evaluation of translation quality is not as expensive or as time consuming as generally thought. We use Amazon’s Mechanical Turk, an online labor market that is designed to pay people small sums of money to complete human intelligence tests – tasks that are difficult for computers but easy for people. We show that: • Non-expert annotators produce judgments that are very similar to experts and that have a stronger correlation than Bleu. • Mechanical Turk can be used for complex tasks like human-mediated translation edit rate (HTER) and creating multiple reference translations. • Evaluating translation quality through read- ing comprehension, which is rarely done, can be easily accomplished through creative use of Mechanical Turk. 2 Related work Snow et al. (2008) examined the accuracy of la- bels created using Mechanical Turk for a variety of natural language processing tasks. These tasks included word sense disambiguation, word simi- larity, textual entailment, and temporal ordering of events, but not machine translation. Snow et al. measured the quality of non-expert annotations by comparing them against labels that had been previously created by expert annotators. They re- port inter-annotator agreement between expert and non-expert annotators, and show that the average of many non-experts converges on performance of a single expert for many of their tasks. Although it is not common for manual evalu- ation results to be reported in conference papers, several large-scale manual …

Content truncated to 500 words