Predicting human translation quality

Predicting Human Translation Quality Lucia Specia l.specia@sheffield.ac.uk Kashif Shah kashif.shah@sheffield.ac.uk Department of Computer Science, University of Sheffield Regent Court, 211 Portobello Street, S4 1DP, UK Abstract We present a first attempt at predicting the quality of translations produced by human, profes- sional translators. We examine datasets annotated for quality at sentence- and word-level for four language pairs and provide experiments with prediction models for these datasets. We compare the performance of such models against that of models built from machine transla- tions, highlighting a number of challenges in estimating quality and detecting errors in human translations. 1 Introduction Metrics for translation quality estimation (QE) (Blatz et al., 2004; Specia et al., 2009) aim at providing an estimate on the quality of a translated text. Such metrics have no access to refer- ence translations, as they are intended for translation systems in use. QE has shown promising results in several applications in the context of Machine Translation (MT), such as improving post-editing efficiency by filtering out low quality segments which would require more effort to correct than translating from scratch (Specia et al., 2009; Specia, 2011), selecting high quality segments to be published as they are, without post-editing (Soricut and Echihabi, 2010), rank- ing or selecting the best translation from multiple MT systems (Specia et al., 2010; Hildebrand and Vogel, 2013; Avramidis, 2013; Avramidis and Popovi´c, 2013), or between translations from either an MT system or a translation memory (He et al., 2010), and highlighting sub-segments that need revision (Bach et al., 2011). Generally speaking, QE models are built using supervised machine learning algorithms from examples of translations at a given granularity level (e.g. sentences). For training, these examples are annotated with quality labels and described by a number of features that can approximate quality (or errors). “Quality” is therefore defined according to the problem at hand and the labelled data, for example, post-editing time for a sentence or word-level errors. For an overview of various algorithms and features we refer the reader to the WMT12-14 shared tasks on QE (Callison-Burch et al., 2012; Bojar et al., 2013, 2014). So far, QE has only been applied to machine translated texts. However, the above men- tioned applications are also valid in the context of human translation. In particular, in scenarios where translations produced by humans may be of variable or questionable levels of reliabil- ity (e.g. crowdsourcing), it becomes important to estimate translation quality to, for example, select among multiple options of human translations (or even a mix of human and machine translations). In addition, even with professionally created translations, quality assurance is a common process and an estimation method could be useful, for example, to sample the lowest quality cases for checking/revision. Even though it is known that human translations are generally different from machine translations, we put forward the hypothesis that it is possible and useful to have automated Al-Onaizan & Simard (Eds.) Proceedings of AMTA 2014, vol. 1: MT Researchers Vancouver, BC © The Authors 288 metrics to estimate …

Content truncated to 500 words