Systematic Comparison of Professional and Crowdsourced Reference Translations for Machine Translation

Proceedings of NAACL-HLT 2013, pages 612–616, Atlanta, Georgia, 9–14 June 2013. c⃝2013 Association for Computational Linguistics Systematic Comparison of Professional and Crowdsourced Reference Translations for Machine Translation Rabih Zbib, Gretchen Markiewicz, Spyros Matsoukas, Richard Schwartz, John Makhoul Raytheon BBN Technologies Cambridge, MA 02138, USA {rzbib,gmarkiew,smatsouk,schwartz,makhoul}@bbn.com Abstract We present a systematic study of the effect of crowdsourced translations on Machine Trans- lation performance. We compare Machine Translation systems trained on the same data but with translations obtained using Amazon’s Mechanical Turk vs. professional translations, and show that the same performance is ob- tained from Mechanical Turk translations at 1/5th the cost. We also show that adding a Me- chanical Turk reference translation of the de- velopment set improves parameter tuning and output evaluation. 1 Introduction Online crowdsourcing services have been shown to be a cheap and effective data annotation resource for various Natural Language Processing (NLP) tasks (Callison-Burch and Dredze, 2010; Zaidan and Callison-Burch, 2011a; Zaidan and Callison-Burch, 2011b). The resulting quality of annotations is high enough to be used for training statistical NLP mod- els, with a saving in cost and time of up to an or- der of magnitude. Statistical Machine Translation (SMT) is one of the NLP tasks that can benefit from crowdsourced annotations. With appropriate quality control mechanisms, reference translations collected by crowdsourcing have been successfully used for training and evaluating SMT systems (Zbib et al., 2012; Zaidan and Callison-Burch, 2011b). In this work, we used Amazon’s Mechanical Turk (MTurk) to obtain alternative reference translations of four Arabic-English parallel corpora previously released by the Linguistic Data Consortium (LDC) for the DARPA BOLT program. This data, totaling over 500K Arabic tokens, was originally collected from web discussion forums and translated profes- sionally to English. We used alternative MTurk translations of the same data to train and evalua- tion MT systems; and conducted the first systematic study that quantifies the effect of the reference trans- lation process on MT output. We found that: • Mechanical Turk can be used to translate enough data for training an MT system at 1/10th the price of professional translation, and at a much faster rate. • Training MT systems on MTurk reference translations gives the same performance as training with professional translations at 20% of the cost. • A second translation of the development set ob- tained via MTurk improves parameter tuning and output evaluation. 2 Previous Work There have been several publications on crowd- sourcing data annotation for NLP. Callison-Burch and Dredze (2010) give an overview of the NAACL- 2010 Workshop on using Mechanical Turk for data annotation. They describe tasks for which MTurk can be used, and summarize a set of best practices. They also include references to the workshop con- tributions. Zaidan and Callison-Burch (2011a) created a monolingual Arabic data set rich in dialectal con- tent from user commentaries on newspaper web- sites. They hired native Arabic speakers on MTurk 612 to identify the dialect level and used the collected la- bels to train automatic dialect identification systems. They did …

Content truncated to 500 words