How do Humans Evaluate Machine Translation

Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 457–466, Lisboa, Portugal, 17-18 September 2015. c⃝2015 Association for Computational Linguistics. How do Humans Evaluate Machine Translation Francisco Guzm´an Ahmed Abdelali Irina Temnikova Hassan Sajjad and Stephan Vogel ALT Research Group Qatar Computing Research Institute, HBKU {fguzman,aabdelali,itemnikova,hsajjad,svogel}@qf.org.qa Abstract In this paper, we take a closer look at the MT evaluation process from a glass-box perspective using eye-tracking. We an- alyze two aspects of the evaluation task – the background of evaluators (monolin- gual or bilingual) and the sources of in- formation available, and we evaluate them using time and consistency as criteria. Our findings show that monolinguals are slower but more consistent than bilinguals, especially when only target language in- formation is available. When exposed to various sources of information, evaluators in general take more time and in the case of monolinguals, there is a drop in consis- tency. Our findings suggest that to have consistent and cost effective MT evalua- tions, it is better to use monolinguals with only target language information. 1 Introduction Each year thousands of human judgments are used to evaluate the quality of Machine Translation (MT) systems to determine which algorithms and techniques are to be considered the new state-of- the-art. In a typical scenario human judges evalu- ate a system’s output (or hypothesis) by comparing it to a source sentence and/or to a reference trans- lation. Then, they score the hypothesis according to a set of defined criteria such as fluency and ad- equacy (White et al., 1994); or rank a set of hy- potheses in order of preference (Vilar et al., 2007; Callison-Burch et al., 2007). Evaluating MT output can be a challenging task for a number of reasons: it is tedious and there- fore evaluators can lose interest quickly; it is com- plex, especially if the guidelines are not well de- fined; and evaluators can have difficulty distin- guishing between different aspects of the transla- tions (Callison-Burch et al., 2007). As a result, evaluations suffer from low inter- and intra-annotator agreements (Turian et al., 2003; Snover et al., 2006). Yet, as Sanders et al. (2011) argue, using human judgments is essential to the progress of MT because: (i) automatic translations are produced for a human audience; and (ii) hu- man understanding of the real world allows to as- sess the importance of the errors made by MT sys- tems. Most of the research in human evaluation has focused on analyzing the criteria to use for evalu- ation, and has regarded the evaluation process as a black-box, where the inputs are different sources of information (i.e source text, reference transla- tion, and translation hypotheses), and the output is a score (or preference ranking). In this paper, we focus on analyzing evalua- tion from a different perspective. First, we regard the process as a glass-box and use eye-tracking to monitor the times evaluators spend digesting different sources of information (scenarios) be- fore making a judgment. Secondly, we contrast how the availability of such sources …

Content truncated to 500 words