Searching for Context: a Study on Document-Level Labels for Translation Quality Estimation

Searching for Context: a Study on Document-Level Labels for Translation Quality Estimation Carolina Scarton1, Marcos Zampieri2,3, Mihaela Vela2, Josef van Genabith2,3 and Lucia Specia1 1University of Sheffield / Regent Court, 211 Portobello, Sheffield, UK 2Saarland University / Campus A2.2, Saarbr¨ucken, Germany 3German Research Centre for Artificial Intelligence / Saarbr¨ucken, Germany {c.scarton,l.specia}@sheffield.ac.uk {marcos.zampieri,m.vela}@uni-saarland.de josef.van genabith@dfki.de Abstract In this paper we analyse the use of pop- ular automatic machine translation evalu- ation metrics to provide labels for qual- ity estimation at document and paragraph levels. We highlight crucial limitations of such metrics for this task, mainly the fact that they disregard the discourse structure of the texts. To better understand these limitations, we designed experiments with human annotators and proposed a way of quantifying differences in translation qual- ity that can only be observed when sen- tences are judged in the context of entire documents or paragraphs. Our results in- dicate that the use of context can lead to more informative labels for quality anno- tation beyond sentence level. 1 Introduction Quality estimation (QE) of machine translation (MT) (Blatz et al., 2004; Specia et al., 2009) is an area that focuses on predicting the quality of new, unseen machine translation data without rely- ing on human references. This is done by training models using features extracted from source and target texts and, when available, from the MT sys- tem, along with a quality label for each instance. Most current work on QE is done at the sentence level. A popular application of sentence-level QE is to support post-editing of MT (He et al., 2010). As quality labels, Likert scores have been used for post-editing effort, as well as post-editing time and edit distance between the MT output and the final version – HTER (Snover et al., 2006). c⃝ 2015 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC- BY-ND. There are, however, scenarios where quality pre- diction beyond sentence level is needed, most no- tably in cases when automatic translations without post-editing are required. This is the case, for ex- ample, of quality prediction for an entire product review translation in order to decide whether or not it can be published as is, so that customers speak- ing other languages can understand it. The quality of a document is often seen as some form of aggregation of the quality of its sentences. We claim, however, that document-level quality assessment should consider more information than sentence-level quality. This includes, for exam- ple, the topic and structure of the document and the relationship between its sentences. While cer- tain sentences are considered perfect in isolation, their combination in context may lead to incoher- ent text. Conversely, while a sentence can be con- sidered poor in isolation, when put in context, it may benefit from information in surrounding sen- tences, leading to a document that is fit for pur- pose. Document-level quality prediction is a rather understudied problem. Recent work has looked into document-level prediction (Scarton and …

Content truncated to 500 words