Exploiting Objective Annotations for Minimising Translation Post-editing Effort

Exploiting Objective Annotations for Measuring Translation Post-editing Effort Lucia Specia Research Group in Computational Linguistics, University of Wolverhampton, Wolverhampton, UK l.specia@wlv.ac.uk Abstract With the noticeable improvement in the overall quality of Machine Translation (MT) systems in recent years, post-editing of MT output is starting to become a com- mon practice among human translators. However, it is well known that the qual- ity of a given MT system can vary sig- nificantly across translation segments and that post-editing bad quality translations is a tedious task that may require more effort than translating texts from scratch. Previ- ous research dedicated to learning quality estimation models to flag such segments has shown that models based on human an- notation achieve more promising results. However, it is not yet clear what is the most appropriate form of human annota- tion for building such models. We exper- iment with models based on three annota- tion types (post-editing time, post-editing distance and post-editing effort scores) and show that estimations resulting from using post-editing time, a simple and objective annotation, can reliably indicate transla- tion post-editing effort in a practical, task- based scenario. We also discuss some per- spectives on the effectiveness, reliability and cost of each type of annotation. 1 Introduction Post-editing Machine Translation (MT) output is now seen as a potentially successful way of incor- porating MT into the human translation workflow in order to minimize time and costs in the transla- tion industry. This is particularly true with Statis- c⃝ 2011 European Association for Machine Translation. tical MT (SMT) systems, which can be built with little effort from translation memories. However, a common complaint from human translators is that the post-editing of certain segments with low qual- ity can be frustrating and can require more effort than translating those segments from scratch, with- out the aid of an MT system. Identifying such seg- ments and filtering them out from the post-editing task is a problem addressed in the field of “Confi- dence Estimation” (CE), also called “Quality Esti- mation”, for MT. CE metrics are usually prediction models in- duced from data using standard machine learn- ing algorithms fed with examples of source and translation features, as well as some form of an- notation on the quality of the translations. Early work on sentence-level CE use annotations derived from automatic MT evaluation metrics (Blatz et al., 2004) such as BLEU (Papineni et al., 2002), NIST (Doddington, 2002) and WER (Tillmann et al., 1997) at training time. The resulting models have not been shown to be effective, since the au- tomatic metrics used do not correlate well with hu- man judgments at the segment level and are dif- ficult to interpret as absolute indicators of qual- ity. More recent work focuses on having humans assigning absolute quality scores to translations, which has shown more promising results (Quirk, 2004; Specia et al., 2009a). Obtaining explicit human annotations for trans- lation quality, i.e., absolute scores reflecting post- editing effort, can however be a time-consuming and subjective task, requiring …

Content truncated to 500 words