The UPC Submission to the WMT 2012 Shared Task on Quality Estimation

Proceedings of the 7th Workshop on Statistical Machine Translation, pages 127–132, Montr´eal, Canada, June 7-8, 2012. c⃝2012 Association for Computational Linguistics The UPC Submission to the WMT 2012 Shared Task on Quality Estimation Daniele Pighin Meritxell Gonz´alez Llu´ıs M`arquez Universitat Polit`ecnica de Catalunya, Barcelona {pighin,mgonzalez,lluism}@lsi.upc.edu Abstract In this paper, we describe the UPC system that participated in the WMT 2012 shared task on Quality Estimation for Machine Translation. Based on the empirical evidence that fluency- related features have a very high correlation with post-editing effort, we present a set of features for the assessment of quality estima- tion for machine translation designed around different kinds of n-gram language models, plus another set of features that model the quality of dependency parses automatically projected from source sentences to transla- tions. We document the results obtained on the shared task dataset, obtained by combining the features that we designed with the baseline features provided by the task organizers. 1 Introduction Quality Estimation (QE) for Machine Translations (MT) is the task concerned with the prediction of the quality of automatic translations in the absence of reference translations. The WMT 2012 shared task on QE for MT (Callison-Burch et al., 2012) required participants to score and rank a set of automatic English to Spanish translations output by a state- of-the-art phrase based machine translation system. Task organizers provided a training dataset of 1, 832 source sentences, together with reference, automatic and post-edited translations, as well as human qual- ity assessments for the automatic translations. Post- editing effort, i.e., the amount of editing required to produce an accurate translation, was selected as the quality criterion, with assessments ranging from 1 (extremely bad) to 5 (good as it is). The organizers also provided a set of linguistic resources and pro- cessors to extract 17 global indicators of translation quality (baseline features) that participants could de- cide to employ for their models. For the evaluation, these features are used to learn a baseline predictors for participants to compare against. Systems partic- ipating in the evaluation are scored based on their ability to correctly rank the 422 test translations (us- ing DeltaAvg and Spearman correlation) and/or to predict the human quality assessment for each trans- lation (using Mean Average Error - MAE and Root Mean Squared Error - RMSE). Our initial approach to the task consisted of sev- eral experiments in which we tried to identify com- mon translation errors and correlate them with qual- ity assessments. However, we soon realized that simple regression models estimated on the baseline features resulted in more consistent predictors of translation quality. For this reason, we eventually decided to focus on the design of a set of global in- dicators of translation quality to be combined with the strong features already computed by the baseline system. An analysis of the Pearson correlation of the baseline features (Callison-Burch et al., 2012)1 with human quality assessments shows that the two strongest individual predictors of post-editing ef- fort are the n-gram language model perplexities es- …

Content truncated to 500 words