Target-Centric Features for Translation Quality Estimation

Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 329–334, Baltimore, Maryland USA, June 26–27, 2014. c⃝2014 Association for Computational Linguistics Target-Centric Features for Translation Quality Estimation Chris Hokamp and Iacer Calixto and Joachim Wagner and Jian Zhang CNGL Centre for Global Intelligent Content Dublin City University School of Computing Dublin, Ireland {chokamp|icalixto|jwagner|zhangj}@computing.dcu.ie Abstract We describe the DCU-MIXED and DCU- SVR submissions to the WMT-14 Quality Estimation task 1.1, predicting sentence- level perceived post-editing effort. Fea- ture design focuses on target-side features as we hypothesise that the source side has little effect on the quality of human trans- lations, which are included in task 1.1 of this year’s WMT Quality Estimation shared task. We experiment with features of the QuEst framework, features of our past work, and three novel feature sets. Despite these efforts, our two systems per- form poorly in the competition. Follow up experiments indicate that the poor perfor- mance is due to improperly optimised pa- rameters. 1 Introduction Translation quality estimation tries to predict the quality of a translation given the source and target text but no reference translations. Different from previous years (Callison-Burch et al., 2012; Bo- jar et al., 2013), the WMT 2014 Quality Estima- tion shared task is MT system-independent, i. e. no glass-box features are available and translations in the training and test sets are produced by different MT systems and also by human translators. This paper describes the CNGL@DCU team submission to task 1.1 of the WMT 2014 Quality Estimation shared task.1 The task is to predict the perceived post-editing effort given a source sen- tence and its raw translation. Due to the inclusion of human translation in the task, we focus our ef- forts on target-side features as we expect that the quality of a translation produced by a human trans- lator is much less affected by features of the source 1A CNGL system based on referential translation ma- chines is submitted separately (Bic¸ici and Way, 2014). than by extrinsic factors such as time pressure and familiarity with the domain. To build our quality estimation system, we use and extend the QuEst framework for translation quality estimation2 (Shah et al., 2013; Specia et al., 2013). QuEst provides modules for fea- ture extraction and machine learning. We modify both the feature extraction framework and the ma- chine learning components to add functionality to QuEst. The novel features we add to our systems are (a) a language model on a combination of stop words and POS tags, (b) inverse glass-box fea- tures for translating the translation, and (c) ran- dom indexing (Sahlgren, 2005) for measuring the semantic similarity of source and target side across languages. Furthermore, we integrated (d) source- side pseudo-reference features (Soricut and Echi- habi, 2010) and (e) error grammar features (Wag- ner, 2012), which were used first in MT quality estimation by (Rubino et al., 2012; Rubino et al., 2013). The remaining sections are organised as fol- lows. Section 2 gives details on the features we use. Section 3 …

Content truncated to 500 words