The Language Demographics of Amazon Mechanical Turk

The Language Demographics of Amazon Mechanical Turk Ellie Pavlick1 Matt Post2 Ann Irvine2 Dmitry Kachaev2 Chris Callison-Burch1,2 1Computer and Information Science Department, University of Pennsylvania 2Human Language Technology Center of Excellence, Johns Hopkins University Abstract We present a large scale study of the languages spoken by bilingual workers on Mechanical Turk (MTurk). We establish a methodology for determining the language skills of anony- mous crowd workers that is more robust than simple surveying. We validate workers’ self- reported language skill claims by measuring their ability to correctly translate words, and by geolocating workers to see if they reside in countries where the languages are likely to be spoken. Rather than posting a one-off survey, we posted paid tasks consisting of 1,000 as- signments to translate a total of 10,000 words in each of 100 languages. Our study ran for several months, and was highly visible on the MTurk crowdsourcing platform, increas- ing the chances that bilingual workers would complete it. Our study was useful both to cre- ate bilingual dictionaries and to act as cen- sus of the bilingual speakers on MTurk. We use this data to recommend languages with the largest speaker populations as good candidates for other researchers who want to develop crowdsourced, multilingual technologies. To further demonstrate the value of creating data via crowdsourcing, we hire workers to create bilingual parallel corpora in six Indian lan- guages, and use them to train statistical ma- chine translation systems. 1 Overview Crowdsourcing is a promising new mechanism for collecting data for natural language processing re- search. Access to a fast, cheap, and flexible work- force allows us to collect new types of data, poten- tially enabling new language technologies. Because crowdsourcing platforms like Amazon Mechanical Turk (MTurk) give researchers access to a world- wide workforce, one obvious application of crowd- sourcing is the creation of multilingual technologies. With an increasing number of active crowd workers located outside of the United States, there is even the potential to reach fluent speakers of lower resource languages. In this paper, we investigate the feasi- bility of hiring language informants on MTurk by conducting the first large-scale demographic study of the languages spoken by workers on the platform. There are several complicating factors when try- ing to take a census of workers on MTurk. The workers’ identities are anonymized, and Amazon provides no information about their countries of ori- gin or their language abilities. Posting a simple sur- vey to have workers report this information may be inadequate, since (a) many workers may never see the survey, (b) many opt not to do one-off surveys since potential payment is low, and (c) validating the answers of respondents is not straightforward. Our study establishes a methodology for deter- mining the language demographics of anonymous crowd workers that is more robust than simple sur- veying. We ask workers what languages they speak and what country they live in, and validate their claims by measuring their ability to correctly trans- late words and by recording their …

Content truncated to 500 words