Cost Optimization in Crowdsourcing Translation: Low cost translations made even cheaper

Human Language Technologies: The 2015 Annual Conference of the North American Chapter of the ACL, pages 705–713, Denver, Colorado, May 31 – June 5, 2015. c⃝2015 Association for Computational Linguistics Cost Optimization for Crowdsourcing Translation Mingkun Gao, Wei Xu and Chris Callison-Burch Computer and Information Science Department University of Pennsylvania, Philadelphia, PA, USA {gmingkun, xwe, ccb}@seas.upenn.edu Abstract Crowdsourcing makes it possible to create translations at much lower cost than hiring professional translators. However, it is still expensive to obtain the millions of transla- tions that are needed to train statistical ma- chine translation systems. We propose two mechanisms to reduce the cost of crowdsourc- ing while maintaining high translation quality. First, we develop a method to reduce redun- dant translations. We train a linear model to evaluate the translation quality on a sentence- by-sentence basis, and fit a threshold between acceptable and unacceptable translations. Un- like past work, which always paid for a fixed number of translations for each source sen- tence and then chose the best from them, we can stop earlier and pay less when we receive a translation that is good enough. Second, we introduce a method to reduce the pool of translators by quickly identifying bad transla- tors after they have translated only a few sen- tences. This also allows us to rank translators, so that we re-hire only good translators to re- duce cost. 1 Introduction Crowdsourcing is a promising new mechanism for collecting large volumes of annotated data at low cost. Many NLP researchers have started creating speech and language data through crowdsourcing (for example, Snow et al. (2008), Callison-Burch and Dredze (2010) and others). One NLP applica- tion that has been the focus of crowdsourced data collection is statistical machine translation (SMT) which requires large bilingual sentence-aligned par- allel corpora to train translation models. Crowd- sourcing’s low cost has made it possible to hire peo- ple to create sufficient volumes of translation in or- der to train SMT systems (for example, Ambati and Vogel (2010), Zbib et al. (2012), Post et al. (2012), Zbib et al. (2013)). However, crowdsourcing is not perfect, and one of its most pressing challenges is how to ensure the quality of the data that is created by it. Unlike in more traditional employment scenarios, where an- notators are pre-vetted and their skills are clear, in crowdsourcing very little is known about the annota- tors. They are not professional translators, and there are no built-in mechanisms for testing their language skills. They complete tasks without any oversight. Thus, translations produced via crowdousrcing may be low quality. Previous work has addressed this problem, showing that non-professional translators hired on Amazon Mechanical Turk (MTurk) can achieve professional-level quality, by soliciting mul- tiple translations of each source sentence and then choosing the best translation (Zaidan and Callison- Burch, 2011). In this paper we focus on a different aspect of crowdsourcing than Zaidan and Callison-Burch (2011). We attempt to achieve the same high qual- ity while minimizing the associated costs. We pro- pose …

Content truncated to 500 words