DCU-Symantec at the WMT 2013 Quality Estimation Shared Task

Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 392–397, Sofia, Bulgaria, August 8-9, 2013 c⃝2013 Association for Computational Linguistics DCU-Symantec at the WMT 2013 Quality Estimation Shared Task Raphael Rubino†⋆, Joachim Wagner†‡, Jennifer Foster†, Johann Roturier⋆ Rasoul Samad Zadeh Kaljahi†⋆ and Fred Hollowood⋆ †NCLT, School of Computing, Dublin City University, Ireland ‡Center for Next Generation Localisation, Dublin, Ireland ⋆Symantec Research Labs, Dublin, Ireland †{rrubino, jwagner, jfoster}@computing.dcu.ie ⋆{johann roturier, fhollowood}@symantec.com Abstract We describe the two systems submit- ted by the DCU-Symantec team to Task 1.1. of the WMT 2013 Shared Task on Quality Estimation for Machine Transla- tion. Task 1.1 involve estimating post- editing effort for English-Spanish trans- lation pairs in the news domain. The two systems use a wide variety of fea- tures, of which the most effective are the word-alignment, n-gram frequency, lan- guage model, POS-tag-based and pseudo- references ones. Both systems perform at a similarly high level in the two tasks of scoring and ranking translations, although there is some evidence that the systems are over-fitting to the training data. 1 Introduction The WMT 2013 Quality Estimation Shared Task involve both sentence-level and word-level qual- ity estimation (QE). The sentence-level task con- sist of three subtasks: scoring and ranking transla- tions with regard to post-editing effort (Task 1.1), selecting among several translations produced by multiple MT systems for the same source sentence (Task 1.2), and predicting post-editing time (Task 1.3). The DCU-Symantec team enter two systems to Task 1.1. Given a set of source English news sentences and their Spanish translations, the goals are to predict the HTER score of each translation and to produce a ranking based on HTER for the set of translations. A set of 2,254 sentence pairs are provided for training. On the ranking task, our system DCU-SYMC alltypes is second placed out of thirteen sys- tems and our system DCU-SYMC combine is ranked fifth, according to the Delta Average met- ric. According to the Spearman rank correlation, our systems are the joint-highest systems. In the scoring task, the DCU-SYMC alltypes system is placed sixth out of seventeen systems accord- ing to Mean Absolute Error (MAE) and third ac- cording to Root Mean Squared Error (RMSE). The DCU-SYMC combine system is placed fifth ac- cording to MAE and second according to RMSE. In this system description paper, we describe the features, the learning methods used, the results for the two submitted systems and some other systems we experiment with. 2 Features Our starting point for the WMT13 QE shared task was the feature set used in the system we submit- ted to the WMT12 QE task (Rubino et al., 2012). This feature set, comprising 308 features in to- tal, extended the 17 baseline features provided by the task organisers to include 6 additional sur- face features, 6 additional language model fea- tures, 17 additional features derived from the MT system components and the n-best lists, 138 features obtained by part-of-speech tagging and parsing the source sentences and 95 obtained by part-of-speech tagging …

Content truncated to 500 words