Comparing human perceptions of post-editing effort with post-editing operations

Proceedings of the 7th Workshop on Statistical Machine Translation, pages 181–190, Montr´eal, Canada, June 7-8, 2012. c⃝2012 Association for Computational Linguistics Comparing human perceptions of post-editing effort with post-editing operations Maarit Koponen University of Helsinki, Dept of Modern Languages PO Box 24 00014 University of Helsinki, Finland maarit.koponen@helsinki.fi Abstract Post-editing performed by translators is an increasingly common use of machine trans- lated texts. While high quality MT may in- crease productivity, post-editing poor transla- tions can be a frustrating task which requires more effort than translating from scratch. For this reason, estimating whether machine trans- lations are of sufficient quality to be used for post-editing and finding means to reduce post-editing effort are an important field of study. Post-editing effort consists of different aspects, of which temporal effort, or the time spent on post-editing, is the most visible and involves not only the technical effort needed to perform the editing, but also the cognitive effort required to detect and plan necessary corrections. Cognitive effort is difficult to ex- amine directly, but ways to reduce the cogni- tive effort in particular may prove valuable in reducing the frustration associated with post- editing work. In this paper, we describe an experiment aimed at studying the relationship between technical post-editing effort and cog- nitive post-editing effort by comparing cases where the edit distance and a manual score re- flecting perceived effort differ. We present re- sults of an error analysis performed on such sentences and discuss the clues they may pro- vide about edits requiring great cognitive ef- fort compared to the technical effort, on one hand, or little cognitive effort, on the other. 1 Introduction An increasingly common use for machine transla- tion is producing texts to be post-edited by transla- tors. While sufficiently high-quality MT has been shown to produce benefits for productivity, a well- known problem is that post-editing poor machine translation can require more effort than translating from scratch. Measuring and estimating post-editing effort is therefore a growing concern addressed by Confidence Estimation (CE) (Specia, 2011). Time spent on post-editing can be seen as the most visible and economically most important as- pect of post-editing effort (Krings, 2001); however, post-editing effort can be defined and approached in different ways. Krings (2001) divides post-editing effort into three types: 1. temporal, 2. cognitive and 3. technical. Temporal effort refers to post- editing time. Cognitive effort involves identifying the errors in the MT and the necessary steps to cor- rect the output. Technical effort then consists of the keystrokes and cut-and-paste operations needed to produce the post-edited version after the errors have been detected and corrections planned. These dif- ferent aspects of effort are not necessarily equal in various situations. In some cases, the errors may be easy to detect but involve several technical opera- tions to be corrected. In other cases, parsing the sen- tence and detecting the errors may require consid- erable cognitive effort, although the actual technical operations required are quick and easy. According to Krings (2001), temporal effort …

Content truncated to 500 words