Linguistic Features for Quality Estimation

Proceedings of the 7th Workshop on Statistical Machine Translation, pages 96–103, Montr´eal, Canada, June 7-8, 2012. c⃝2012 Association for Computational Linguistics Linguistic Features for Quality Estimation Mariano Felice Research Group in Computational Linguistics University of Wolverhampton Stafford Street Wolverhampton, WV1 1SB, UK Mariano.Felice@wlv.ac.uk Lucia Specia Department of Computer Science University of Sheffield Regent Court, 211 Portobello Sheffield, S1 4DP, UK L.Specia@dcs.shef.ac.uk Abstract This paper describes a study on the contribu- tion of linguistically-informed features to the task of quality estimation for machine trans- lation at sentence level. A standard regression algorithm is used to build models using a com- bination of linguistic and non-linguistic fea- tures extracted from the input text and its ma- chine translation. Experiments with English- Spanish translations show that linguistic fea- tures, although informative on their own, are not yet able to outperform shallower features based on statistics from the input text, its translation and additional corpora. However, further analysis suggests that linguistic infor- mation is actually useful but needs to be care- fully combined with other features in order to produce better results. 1 Introduction Estimating the quality of automatic translations is becoming a subject of increasing interest within the Machine Translation (MT) community for a num- ber of reasons, such as helping human translators post-editing MT, warning users about non-reliable translations or combining output from multiple MT systems. Different from most classic approaches for measuring the progress of an MT system or compar- ing MT systems, which assess quality by contrast- ing system output to reference translations such as BLEU (Papineni et al., 2002), Quality Estimation (QE) is a more challenging task, aimed at MT sys- tems in use, and therefore without access to refer- ence translations. From the findings of previous work on reference- dependent MT evaluation, it is clear that metrics exploiting linguistic information can achieve sig- nificantly better correlation with human judgments on quality, particularly at the level of sentences (Gim´enez and M`arquez, 2010). Intuitively, this should also apply for quality estimation metrics: while evaluation metrics compare linguistic repre- sentations of the system output and reference trans- lations (e.g. matching of n-grams of part-of-speech tags or predicate-argument structures), quality esti- mation metrics would perform the (more complex) comparison og linguistic representations of the input and translation texts. The hypothesis put forward in this paper is therefore that using linguistic informa- tion to somehow contrast the input and translation texts can be beneficial for quality estimation. We test this hypothesis as part of the WMT-12 shared task on quality estimation. The system sub- mitted to this task (WLV-SHEF) integrates linguis- tic information to a strong baseline system using only shallow statistics from the input and transla- tion texts, with no explicit information from the MT system that produced the translations. A variant also tests the addition of linguistic information to a larger set of shallow features. The quality esti- mation problem is modelled as a supervised regres- sion task using Support Vector Machines (SVM), which has been shown to achieve good performance …

Content truncated to 500 words