The SDL Language Weaver Systems in the WMT12 Quality Estimation Shared Task

Proceedings of the 7th Workshop on Statistical Machine Translation, pages 145–151, Montr´eal, Canada, June 7-8, 2012. c⃝2012 Association for Computational Linguistics The SDL Language Weaver Systems in the WMT12 Quality Estimation Shared Task Radu Soricut Nguyen Bach SDL Language WeaverZiyuan Wang 6060 Center Drive, Suite 101 Los Angeles, CA, USA {rsoricut,nbach,zwang}@sdl.com Abstract We present in this paper the system sub- missions of the SDL Language Weaver team in the WMT 2012 Quality Estimation shared-task. Our MT quality-prediction sys- tems use machine learning techniques (M5P regression-tree and SVM-regression models) and a feature-selection algorithm that has been designed to directly optimize towards the of- ficial metrics used in this shared-task. The resulting submissions placed 1st (the M5P model) and 2nd (the SVM model), respec- tively, on both the Ranking task and the Scor- ing task, out of 11 participating teams. 1 Introduction The WMT 2012 Quality Estimation shared-task fo- cused on automatic methods for estimating machine translation output quality at run-time (sentence-level estimation). Different from MT evaluation met- rics, quality prediction (QP) systems do not rely on reference translations and are generally built us- ing machine learning techniques to estimate quality scores (Specia et al., 2009; Soricut and Echihabi, 2010; Bach et al., 2011; Specia, 2011). Some interesting uses of sentence-level MT qual- ity prediction are the following: decide whether a given translation is good enough for publishing as- is (Soricut and Echihabi, 2010), or inform monolin- gual (target-language) readers whether or not they can rely on a translation; filter out sentences that are not good enough for post-editing by professional translators (Specia, 2011); select the best translation among options from multiple MT systems (Soricut and Narsale, 2012), etc. This shared-task focused on estimating the qual- ity of English to Spanish automatic translations. The training set distributed for the shared task comprised of 1, 832 English sentences taken from the news do- main and their Spanish translations. The translations were produced by the Moses SMT system (Koehn et al., 2007) trained on Europarl data. Translations also had a quality score derived from an average of three human judgements of Post-Editing effort using a 1- 5 scale (1 for worse-quality/most-effort, and 5 for best-quality/least-effort). Submissions were evalu- ated using a blind official test set of 422 sentences produced in the same fashion as the training set. Two sub-tasks were considered: (i) scoring transla- tions using the 1-5 quality scores (Scoring), and (ii) ranking translations from best to worse (Ranking). The official metrics used for the Ranking task were DeltaAvg (measuring how valuable a proposed rank- ing is from the perspective of extrinsic values asso- ciated with the test entries, in this case post-editing effort on a 1-5 scale; for instance, a DeltaAvg of 0.5 means that the top-ranked quantiles have +0.5 bet- ter quality on average compared to the entire set), as well as the Spearman ranking correlation. For the Scoring task the metrics were Mean-Absolute-Error (MAE) and Root Mean Squared Error (RMSE). The interested reader is referred to (Callison-Burch et al., 2012) …

Content truncated to 500 words