Relations between different types of post-editing operations, cognitive effort and temporal effort

Relations between different types of post-editing operations, cognitive effort and temporal effort Maja Popovi´c, Arle Lommel, Aljoscha Burchardt, Eleftherios Avramidis, Hans Uszkoreit DFKI – Berlin, Germany name.surname@dfki.de Abstract Despite the growing interest in and use of machine translation post-edited out- puts, there is little research work explor- ing different types of post-editing opera- tions, i.e. types of translation errors cor- rected by post-editing. This work in- vestigates five types of post-edit oper- ations and their relation with cognitive post-editing effort (quality level) and post- editing time. Our results show that for French-to-English and English-to-Spanish translation outputs, lexical and word or- der edit operations require most cogni- tive effort, lexical edits require most time, whereas removing additions has a low im- pact both on quality and on time. It is also shown that the sentence length is an impor- tant factor for the post-editing time. 1 Introduction and related work In machine translation research, ever-increasing amounts of post-edited translation outputs are be- ing collected. These have been used primarily for automatic estimation of translation quality. How- ever, they enable a large number of applications, such as analysis of different aspects of post-editing effort. (Krings, 2001) defines three aspects: tem- poral, referring to time spent on post-editing, cog- nitive, referring to identifying the errors and the necessary steps for correction, and technical, refer- ring to edit operations performed in order to pro- duce the post-edited version. These aspects of ef- fort are not necessary equal in various situations. c⃝ 2014 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC- BY-ND. Since the temporal aspect is important for the practice, post-editing time is widely used for mea- suring post-editing effort (Krings, 2001; Tatsumi, 2009; Tatsumi et Roturier, 2010; Specia, 2011). Human quality scores based on the needed amount of post-editing are involved as assessment of the cognitive effort in (Specia et al., 2010; Specia, 2011). Using edit distance between the original and the post-edited translation for assessment of the technical effort is reported in (Tatsumi, 2009; Tatsumi et Roturier, 2010; Temnikova, 2010; Spe- cia, 2011; Blain et al., 2011). More details about the technical effort can be obtained by analysing particular edit operations. (Blain et al., 2011) defined these operations on a linguistic level as post-editing actions and per- formed comparison between statistical and rule- based systems. (Temnikova, 2010) proposed the analysis of edit operations for controlled language in order to explore cognitive effort for different error types – post-editors assigned one of ten er- ror types to each edit operation which were then ranked by difficulty. In (Koponen, 2012) post-edit operations are analysed in sentences with discrep- ancy between the assigned quality score and the number of performed post-edits. In one of the ex- periments described in (Wisniewski et al., 2013) an automatic analysis of post-edits based on Lev- enshtein distance is carried out considering only the basic level of substitutions, deletions, inser- tions and TER shifts. These edit operations are analysed on the …

Content truncated to 500 words