Quality estimation for translation selection

Quality estimation for translation selection Kashif Shah and Lucia Specia Department of Computer Science University of Sheffield S1 4DP, UK {kashif.shah,l.specia}@sheffield.ac.uk Abstract We describe experiments on quality esti- mation to select the best translation among multiple options for a given source sen- tence. We consider a realistic and chal- lenging setting where the translation sys- tems used are unknown, and no relative quality assessments are available for the training of prediction models. Our findings indicate that prediction errors are higher in this blind setting. However, these errors do not have a negative impact in performance when the predictions are used to select the best translation, compared to non-blind settings. This holds even when test condi- tions (text domains, MT systems) are dif- ferent from model building conditions. In addition, we experiment with quality pre- diction for translations produced by both translation systems and human translators. Although the latter are on average of much higher quality, we show that automatically distinguishing the two types of translation is not a trivial problem. 1 Introduction Quality Estimation (QE) [Blatz et al., 2004, Specia et al., 2009] has several applications in the context of Machine Translation (MT), considering the use of translations for both inbound (e.g. gisting) and outbound (e.g. post-editing) purposes. To date, research on quality estimation has been focus- ing mostly on predicting absolute single-sentence quality scores. However, for certain applications c⃝ 2014 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC- BY-ND. an absolute score may not be necessary. Our goal is to model quality estimation by contrasting the output of several translation sources for the same input sentence against each other. The outcome of this process is a ranking of alternative transla- tions based on their predicted quality. For our ap- plication, we are only interested in the top-ranked translation, which could for example be provided to a human post-editor for revision. Previous research on this task has focused on ranking translations from multiple MT systems where system identifiers are known beforehand. Based on such identifiers, individual quality pre- diction models can be trained for each MT system [Specia et al., 2010], and the predicted (absolute) scores for translations of a given source sentence across multiple MT systems used to rank them. Al- ternatively, quality prediction models can be built to directly output a ranking of alternative transla- tions based on training data annotated with relative quality scores, using for example pairwise rank- ing algorithms [Avramidis, 2013, Avramidis and Popovi´c, 2013]. In this paper we model translation selection con- sidering a scenario where translations are produced by multiple MT systems, but the identifiers of the MT systems are not given, i.e., we assume a blind setting where the sources of the translations are not known. While ranking approaches to system selection could also be used in this blind setting, they require training data labelled with compara- tive assessments of translations produced by mul- tiple sources. In our experiments, we assume a more …

Content truncated to 500 words