Phrase Level Segmentation and Labelling of Machine Translation Errors

Phrase-Level Segmentation and Labelling of Machine Translation Errors Fr´ed´eric Blain, Varvara Logacheva, Lucia Specia Department of Computer Science University of Sheffield Sheffield, UK {f.blain,v.logacheva,l.specia}@sheffield.ac.uk Abstract This paper presents our work towards a novel approach for Quality Estimation (QE) of machine translation based on sequences of adjacent words, the so-called phrases. This new level of QE aims to provide a natural balance between QE at word and sentence-level, which are either too fine grained or too coarse levels for some applications. However, phrase-level QE implies an intrinsic challenge: how to segment a machine translation into sequence of words (contiguous or not) that represent an error. We discuss three possible segmentation strategies to automatically extract erroneous phrases. We evaluate these strategies against annotations at phrase-level produced by humans, using a new dataset collected for this purpose. Keywords: Machine Translation, Post-Editing, Quality Estimation 1. Introduction We recently started to investigate Quality Estimation (QE) for Machine Translation (MT) at phrase-level (Logacheva and Specia, 2015) as a way to balance between word and sentence-level prediction, two well studied levels. Sentence-level QE generally aims to predict if a translation is either good enough or needs to be edited (and sometimes how much editing it needs). This is too coarse for certain tasks, for example, highlighting errors that need to be fixed. Word-level QE can help post-editors by highlighting words with errors, however, it is often hard to predict if an in- dividual word is erroneous. Errors are generally intercon- nected within a segment, and it would be more beneficial for a post-editor if words belonging to the same instance of error could be grouped together, particularly for discontin- uous errors, such as words in incorrect positions. However, contrary to the word-level QE, for which the segmentation boundaries are self-defined and clear, QE at phrase-level implies that one needs to delimit sub-segments within the segment. This is not a trivial task as several alternatives can be used to define a phrase, but in our case the segmentation needs to be connected to the errors in the translation. QE at the phrase-level can reduce human post-editing ef- fort by pinpointing the erroneous sequences that need to be fixed by the post-editor. It can also support automatic post-editing systems (McKeown et al., 2012; Chatterjee et al., 2015), by limiting the post-editing to sequences pre- dicted as incorrect, and thus preventing risky edits that can make the translation even worse. The interest for automatic phrase-level segmentation (and labelling) is however not limited to QE. Given a human post-edition and its origi- nal machine translation, the combination of a monolingual alignment technique with an appropriate phrase segmenta- tion would allow a more detailed analysis of the translation errors. In Section 2. we discuss three possible ways of automat- ically segmenting a translation into phrases and labelling them with binary labels: “OK” or “BAD”, with the latter in- dicating an error. In order to evaluate these strategies, we propose a new gold-standard resource built based on human annotations. The details of both …

Content truncated to 500 words