Eyes Don’t Lie: Predicting Machine Translation Quality Using Eye Movement

Proceedings of NAACL-HLT 2016, pages 1082–1088, San Diego, California, June 12-17, 2016. c⃝2016 Association for Computational Linguistics Eyes Don’t Lie: Predicting Machine Translation Quality Using Eye Movement Hassan Sajjad, Francisco Guzm´an, Nadir Durrani, Ahmed Abdelali, Houda Bouamor†, Irina Temnikova, Stephan Vogel Qatar Computing Research Institute, HBKU, Qatar †Carnegie Mellon University, Qatar Abstract Poorly translated text is often disfluent and difficult to read. In contrast, well-formed translations require less time to process. In this paper, we model the differences in reading patterns of Machine Translation (MT) evalua- tors using novel features extracted from their gaze data, and we learn to predict the qual- ity scores given by those evaluators. We test our predictions in a pairwise ranking scenario, measuring Kendall’s tau correlation with the judgments. We show that our features provide information beyond fluency, and can be com- bined with BLEU for better predictions. Fur- thermore, our results show that reading pat- terns can be used to build semi-automatic met- rics that anticipate the scores given by the evaluators. 1 Introduction Human evaluation has been the preferred method for tracking the progress of MT systems. In the past, the prevalent criterion was to judge the quality of a translation in terms of fluency and adequacy, on an absolute scale (White et al., 1994). However, dif- ferent evaluators focused on different aspects of the translations, which increased the subjectivity of their judgments. As a result, evaluations suffered from low inter- and intra-annotator agreements (Turian et al., 2003; Snover et al., 2006). This caused a shift to- wards a ranking-based approach (Callison-Burch et al., 2007). Unfortunately, the disagreement between evaluators is still a challenge that cannot be easily resolved due to the non-transparent thought-process that evaluators follow to make a judgment. The eye-mind hypothesis (Just and Carpenter, 1980; Potter, 1983) states that when completing a task, people cognitively process objects that are in front of their eyes (i.e. where they fixate their gaze).1 Based on this assumption, it has been possible to study reading behavior and patterns (Rayner, 1998; Garrod, 2006; Hansen and Ji, 2010). The overall difficulty of a sentence and its syn- tactic complexity affects reading behavior (Coco and Keller, 2015). Ill-formed sentences take longer to process, and may cause the reader to jump back while reading. Hence, by looking into how evalu- ators read the translations and their accompanying references, we can learn about: (i) the complexity of a reference sentence, and (ii) the quality of a trans- lation sentence. Using reading patterns from evaluators could be a useful tool for MT evaluation: (i) to shed light into the evaluation process: e.g. the general reading be- havior that evaluators follow to complete their task; (ii) to understand which parts of a translation are more difficult for the annotator; and (iii) to develop semi-automatic evaluation systems that use reading patterns to predict translation quality. In this paper, we make a first step towards (iii): us- ing reading patterns as a method for distinguishing between good and bad translations. Our hypothesis is …

Content truncated to 500 words