Applying the Cognitive Machine Translation Evaluation Approach to Arabic

Applying the Cognitive Machine Translation Evaluation Approach to Arabic Irina Temnikova, Wajdi Zaghouani†, Stephan Vogel, Nizar Habash‡ Qatar Computing Research Institute, HBKU †Carnegie Mellon University in Qatar, ‡New York University Abu Dhabi itemnikova@qf.org.qa, wajdiz@cmu.edu, svogel@qf.org.qa, nizar.habash@nyu.edu Abstract The goal of the cognitive machine translation (MT) evaluation approach is to build classifiers which assign post-editing effort scores to new texts. The approach helps estimate fair compensation for post-editors in the translation industry by evaluating the cognitive difficulty of post-editing MT output. The approach counts the number of errors classified in different categories on the basis of how much cognitive effort they require in order to be corrected. In this paper, we present the results of applying an existing cognitive evaluation approach to Modern Standard Arabic (MSA). We provide a comparison of the number of errors and categories of errors in three MSA texts of different MT quality (without any language-specific adaptation), as well as a comparison between MSA texts and texts from three Indo-European languages (Russian, Spanish, and Bulgarian), taken from a previous experiment. The results show how the error distributions change passing from the MSA texts of worse MT quality to MSA texts of better MT quality, as well as a similarity in distinguishing the texts of better MT quality for all four languages. Keywords: Arabic, machine translation evaluation, post-editing 1. Introduction Machine Translation (MT) today is used more and more by professional translators, including freelancers, companies, and official organisations, such as, for example, the Euro- pean Parliament. MT output, especially of publicly avail- able MT engines, such as Google Translate,1 is, however, well known to contain errors and lack fluency from human expectations’ point of view. For this reason, the MT trans- lated texts often need manual (or automatic) corrections, known as “Post-Editing” (PE) (Allen, 2001; Somers, 2003). Although there are fast and simple measures of post-editing cost, such as time to post-edit, or edit-distance, these mea- sures do not reflect the cognitive difficulty involved in cor- recting the specific errors in the MT output text. As the MT output texts can be of different quality and thus con- tain errors of different difficulty to be corrected, fair com- pensation of post-editing should take into account the dif- ficulty of the task, which should thus be measured in the most reliable way. The best solution for this would be to build an automatic classifier which (a) assigns each MT error into a specific correction class, (b) assigns an effort value which reflects the cognitive effort a post-editor needs to make in order to make such a correction, and (c) gives a post-editing effort score to a text. On our way of building such a classifier, we investigate whether an existing cogni- tive effort model (Temnikova, 2010) could provide a fairer compensation for the post-editor, by testing it on a new lan- guage which strongly differs from the previous languages on which this methodology was tested. The model made use of the Statistical Machine Translation (SMT) error classification schema proposed …

Content truncated to 500 words