IRT-based Aggregation Model of Crowdsourced Pairwise Comparison for Evaluating Machine Translations

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 511–520, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics IRT-based Aggregation Model of Crowdsourced Pairwise Comparisons for Evaluating Machine Translations Naoki Otani1 Toshiaki Nakazawa2 Daisuke Kawahara1 Sadao Kurohashi1 1Graduate School of Informatics, Kyoto University, Yoshida-honmachi, Sakyo-ku, Kyoto, Japan 2Japan Science and Technology Agency, Kawaguchi-shi, Saitama, Japan otani.naoki.65v@st.kyoto-u.ac.jp nakazawa@pa.jst.jp {dk,kuro}@i.kyoto-u.ac.jp Abstract Recent work on machine translation has used crowdsourcing to reduce costs of manual eval- uations. However, crowdsourced judgments are often biased and inaccurate. In this pa- per, we present a statistical model that ag- gregates many manual pairwise comparisons to robustly measure a machine translation system’s performance. Our method applies graded response model from item response theory (IRT), which was originally developed for academic tests. We conducted experi- ments on a public dataset from the Workshop on Statistical Machine Translation 2013, and found that our approach resulted in highly in- terpretable estimates and was less affected by noisy judges than previously proposed meth- ods. 1 Introduction Manual evaluation is a primary means of interpret- ing the performance of machine translation (MT) systems and evaluating the accuracy of automatic evaluation metrics. It is also essential for natural lan- guage processing tasks such as summarization and dialogue systems, where (1) the number of correct outputs is unlimited, and (2) na¨ıve text matching cannot judge the correctness, that is, an evaluator must consider syntactic and semantic information. Recent work has used crowdsourcing to reduce costs of manual evaluations. However, the judg- ments of crowd workers are often noisy and unre- liable because they are not experts. To maintain quality, evaluation tasks imple- mented using crowdsourcing should be simple. Thus, many previous studies focused on pairwise comparisons instead of absolute evaluations. The same task is given to multiple workers, and their re- sponses are aggregated to obtain a reliable answer. We must, therefore, develop methods that ro- bustly estimate the MT performance based on many pairwise comparisons. Some aggregation methods have been proposed for MT competitions hosted by the Workshop on Statistical Machine Translation (WMT) (Bojar et al., 2013; Hopkins and May, 2013; Sakaguchi et al., 2014), where a ranking of the submitted systems is produced by aggregating many manual judgments of pairwise comparisons of system outputs. However, existing methods do not consider the following important issues. Interpretability of the estimates: For the purpose of evaluation, their results must be interpretable so that we could use the results to improve MT systems and the next MT evaluation campaigns. Existing methods, however, only yield system-level scores. Judge sensitivity: Some judges can examine the quality of translations with consistent standards, but others cannot (Graham et al., 2015). Sensitivities to the translation quality and judges’ own standards are important factors. Evaluation of a newly submitted system: Pre- vious approaches considered all pairwise combina- tions of systems and must compare a newly sub- mitted system with all the submitted systems. This made it difficult to allow participants to submit their systems after starting …

Content truncated to 500 words