PE2rr Corpus: Manual Error Annotation of Automatically Pre-annotated MT Post-edits

PE2rr Corpus: Manual Error Annotation of Automatically Pre-annotated MT Post-edits Maja Popovi´c Mihael Arˇcan Humboldt University of Berlin Insight Centre for Data Analytics Berlin, Germany NUI Galway, Ireland maja.popovic@hu-berlin.de mihael.arcan@insight-centre.org Abstract We present a freely available corpus containing source language texts from different domains along with their automatically generated translations into several distinct morphologically rich languages, their post-edited versions, and error annotations of the performed post-edit operations. We believe that the corpus will be useful for many different applications. The main advantage of the approach used for creation of the corpus is the fusion of post-editing and error classification tasks, which have usually been seen as two independent tasks, although naturally they are not. We also show benefits of coupling automatic and manual error classification which facilitates the complex manual error annotation task as well as the development of automatic error classification tools. In addition, the approach facilitates annotation of language pair related issues. Keywords: machine translation, post-editing, error annotation 1. Introduction The amount of post-edited translation outputs as well as the amount of data containing annotated machine translation errors is growing in the recent years, with more and more applications and growing need for such data. The process of improving a machine-generated translation is a natural task – post-edited translations are a by-product of the pro- fessional translation workflow, whereas (explicit) transla- tion error classification, which is a very difficult task, is not. On the other hand, the two tasks are actually highly related – post-editing can be viewed as implicit error annotation, since each edit operation is actually a correction of a trans- lation error. Despite of this fact, these two tasks have almost always been seen and performed completely separated and independent. Therefore, data containing error annotations of actual post-edit operations are scarce. The corpus PE2rr aims to improve this situation, containing post-edited trans- lation outputs where each post-edit operation is assigned to a particular error class. In addition, another discrepance in the error classification experiments and corpora can be observed: manual error classification and automatic error classification have always been carried out completely separately from each other. Whereas the results of manual error annotation present a valuable resource for development of automatic error clas- sification tools, this independent approach makes detailed analysis of drawbacks of automatic tools difficult. The PE2rr corpus has been created by merging these two tasks together: automatic error classification has been performed as a first step, and the manual error classification has been performed as a correction of these automatically pre- annotated errors. A very important advantage of this ap- proach is also reducing the effort of the complex and time- consuming manual error classification process, since cor- recting pre-annotated errors is easier and faster than looking for the errors and defining them from scratch. We believe that the approach can also improve the inter-annotator (and intra-annotator) agreement, however this aspect will be in- vestigated in future work. The benefits for the community can be manifold: • facilitating improvement and development of …

Content truncated to 500 words