Estimating post-editing effort: a study on human judgements, task-based and reference-based metrics of MT quality

Estimating post-editing effort: a study on human judgements, task-based and reference-based metrics of MT quality Carolina Scarton1, Mikel L. Forcada2, Miquel Espl`a-Gomis2, Lucia Specia1,3 1 Department of Computer Science, University of Sheffield, Sheffield S1 4DP, UK 2 Dept. Llenguatges i Sist. Inform., Universitat d’Alacant, 03690 St. Vicent del Raspeig, Spain 3 Department of Computing, Imperial College London, London SW7 2AZ, UK. c.scarton@sheffield.ac.uk, mlf@ua.es, mespla@dlsi.ua.es, l.specia@imperial.ac.uk Abstract Devising metrics to assess translation quality has always been at the core of machine translation (MT) research. Tra- ditional automatic reference-based metrics, such as BLEU, have shown correlations with human judgements of ade- quacy and fluency and have been paramount for the advance- ment of MT system development. Crowd-sourcing has popu- larised and enabled the scalability of metrics based on human judgments, such as subjective direct assessments (DA) of ad- equacy, that are believed to be more reliable than reference- based automatic metrics. Finally, task-based measurements, such as post-editing time, are expected to provide a more de- tailed evaluation of the usefulness of translations for a spe- cific task. Therefore, while DA averages adequacy judge- ments to obtain an appraisal of (perceived) quality indepen- dently of the task, and reference-based automatic metrics try to objectively estimate quality also in a task-independent way, task-based metrics are measurements obtained either during or after performing a specific task. In this paper we argue that, although expensive, task-based measurements are the most reliable when estimating MT quality in a specific task; in our case, this task is post-editing. To that end, we report experiments on a dataset with newly-collected post- editing indicators and show their usefulness when estimating post-editing effort. Our results show that task-based metrics comparing machine-translated and post-edited versions are the best at tracking post-editing effort, as expected. These metrics are followed by DA, and then by metrics comparing the machine-translated version and independent references. We suggest that MT practitioners should be aware of these differences and acknowledge their implications when decid- ing how to evaluate MT for post-editing purposes. 1. Introduction Assessing the quality of the output of machine translation (MT) systems has been a widely explored topic in the last two decades. As with other applications outputting language (e.g. text summarisation), quality assessment of MT is chal- lenging and highly dependent on the purpose of the transla- tion. Therefore, the quality of machine translated (MT’ed) texts may depend on their usage. Table 1 shows two machine translations into English, and their respective post-edited ver- sions. In Example 1, the MT’ed version has a different mean- ing from that of the original sentence: readers may be led to believe that the product is good and they should buy it, whilst the correct recommendation is against buying the product. Although this sentence would be problematic for an end user, it is rather simple to correct by a post-editor (only one word needs to be added). Example 2, on the other hand, shows a sentence where an end user can understand the MT’ed ver- sion with little …

Content truncated to 500 words