Crowd-based MT Evaluation for non-English Target Languages

Crowd-based MT Evaluation for non-English Target Languages Michael Paul and Eiichiro Sumita NICT Hikaridai 3-5 619-0289 Kyoto, Japan <Firstname>.<Lastname>@nict.go.jp Luisa Bentivogli and Marcello Federico FBK-irst Via Sommarive, 18 38123 Povo-Trento, Italy {bentivo,federico}@fbk.eu Abstract This paper investigates the feasibility of using crowd-sourcing services for the hu- man assessment of machine translation quality of translations into non-English tar- get languages. Non-expert graders are hired through the CrowdFlower interface to Amazon’s Mechanical Turk in order to carry out a ranking-based MT evaluation of utterances taken from the travel conver- sation domain for 10 Indo-European and Asian languages. The collected human as- sessments are analyzed for their worker characteristics, evaluation costs, and qual- ity of the evaluations in terms of the agree- ment between non-expert graders and ex- pert/oracle judgments. Moreover, data quality control mechanisms including “lo- cale qualification” “qualificatio testing”, and “on-the-fl verification are investi- gated in order to increase the reliability of the crowd-based evaluation results. 1 Introduction This paper focuses on the evaluation of machine translation (MT) quality for target languages other than English. Although human evaluation of MT output provides the most direct and reliable as- sessment, it is time consuming, costly, and subjec- tive. Various automatic evaluation measures were proposed to make the evaluation of MT outputs cheaper and faster (Przybocki et al., 2008), but au- tomatic metrics have not yet proved able to con- sistently predict the usefulness of MT technolo- gies. To counter the high costs in human assess- ment of MT outputs, the usage of crowdsourc- ing services such as Amazon’s Mechanical Turk1 (MTurk) and CrowdFlower2 (CF) were proposed recently (Callison-Burch, 2009; Callison-Burch et al., 2010; Denkowski and Lavie, 2010). 1http://www.mturk.com 2http://crowdfl wer.com The feasibility of crowd-based MT evaluations was investigated for shared tasks such as the WMT (Callison-Burch, 2009) and the IWSLT (Federico et al., 2011) evaluation campaigns. Their re- sults showed that agreement rates for non-experts were comparable to those for experts, and that the crowd-based rankings correlated very strongly with the expert-based rankings. Most of the crowd-based evaluation experiments focused on English as the target language, with the exception of (Callison-Burch et al., 2010) evaluating Czech, French, German, and Spanish translation outputs and (Federico et al., 2011) evaluating translations into French. This paper investigates the feasibility of using crowdsourcing services for the human assessment of translation quality of translation tasks where the target language is not English, with a focus on non-European languages. In order to identify non- English target languages for which we can expect to fin qualifie workers, we referred to existing surveys that analyze the demographics of MTurk workers (see Section 2). In total, we selected 7 non-European languages consisting of Arabic (ar), Chinese (zh), Hindi (hi), Japanese (ja), Korean (ko), Russian (ru), and Tagalog (tl), as well as 3 European languages covering English (en), French (fr), and Spanish (es) as the target languages for our translation experiments. The MT evaluation was carried out using utter- ances taken from the domain of travel conversa- tions. A description of the utilized …

Content truncated to 500 words