Building a Corpus of Errors and Quality in Machine Translation: Experiments on Error Impact

Building a Corpus of Errors and Quality in Machine Translation: Experiments on Error Impact ˆAngela Costa, Rui Correia, Lu´ısa Coheur INESC-ID and CLUNL, INESC-ID and LTI/CMU, INESC-ID and IST Rua Alves Redol, 9 1000-029 Lisboa angela@l2f.inesc-id.pt, rui.correia@l2f.inesc-id.pt, luisa.coheur@inesc-id.pt Abstract In this paper we describe a corpus of automatic translations annotated with both error type and quality. The 300 sentences that we have selected were generated by Google Translate, Systran and two in-house Machine Translation systems that use Moses technology. The errors present on the translations were annotated with an error taxonomy that divides errors in five main linguistic categories (Orthography, Lexis, Grammar, Semantics and Discourse), reflecting the language level where the error is located. After the error annotation process, we accessed the translation quality of each sentence using a four point comprehension scale from 1 to 5. Both tasks of error and quality annotation were performed by two different annotators, achieving good levels of inter-annotator agreement. The creation of this corpus allowed us to use it as training data for a translation quality classifier. We concluded on error severity by observing the outputs of two machine learning classifiers: a decision tree and a regression model. Keywords: machine translation, translation errors, translation quality 1. Introduction Error analysis is a linguistic discipline that has been mostly connected to language learning, but more recently has been used to evaluate automatic language production, like Speech Recognition, Summarization, but specially in Ma- chine Translation (MT). In this paper, we describe the creation of a corpus annotated with errors and translation quality. First, using a specifi- cally designed taxonomy (Costa et al., 2015), we annotate the errors present on translations generated by four differ- ent systems: two mainstream online translation systems – Google Translate (Statistical) and Systran (Hybrid Machine Translation) – and two in-house MT systems that use Moses technology. This is done in three scenarios representing different translation challenges from English (EN) to Euro- pean Portuguese (EP). We then evaluate translation quality using a 4-level classification system based on comprehen- sion. The motivation to build a corpus annotated with errors and translation quality was to help determine future research di- rections in the MT area, allowing, for instance, to highlight which kind of errors hinder comprehension the most. Finally, for this paper, using the presented corpus and the types of errors from the taxonomy of choice as features, we set up different machine learning classifiers to predict translation quality (decision trees and regression models). Then, by looking into the outputs and parameters of the classifiers, we infer the severity of different error types and show how their presence impacts on quality. In Section 2. we present previous work on error annota- tion and translation quality. Section 3. describes the text sources, the tools and the annotation process (errors and quality). Section 4. addresses error gravity, showing the classification results. Finally, in Section 5., we highlight the main conclusions and point to future work directions. 2. Related work Several studies have been developed with the goal …

Content truncated to 500 words