Quality estimation for Machine Translation output using linguistic analysis and decoding features

Proceedings of the 7th Workshop on Statistical Machine Translation, pages 84–90, Montr´eal, Canada, June 7-8, 2012. c⃝2012 Association for Computational Linguistics Quality Estimation for Machine Translation output using linguistic analysis and decoding features Eleftherios Avramidis German Research Center for Artificial Intelligence (DFKI) Berlin, Germany eleftherios.avramidis@dfki.de Abstract We describe a submission to the WMT12 Quality Estimation task, including an exten- sive Machine Learning experimentation. Data were augmented with features from linguis- tic analysis and statistical features from the SMT search graph. Several Feature Selec- tion algorithms were employed. The Quality Estimation problem was addressed both as a regression task and as a discretised classifi- cation task, but the latter did not generalise well on the unseen testset. The most success- ful regression methods had an RMSE of 0.86 and were trained with a feature set given by Correlation-based Feature Selection. Indica- tions that RMSE is not always sufficient for measuring performance were observed. 1 Introduction As Machine Translation (MT) gradually gains a po- sition into production environments, the need for es- timating the quality of its output is increasing. Vari- ous use cases refer to it as input assessment for Hu- man Post-editing, as an extension for Hybrid MT or System Combination, or even a method for improv- ing components of existing MT systems. With the current submission we are trying to address the problem of assigning a quality score to a single MT output per source sentence. Pre- vious work includes regression methods for in- dicating a binary value of correctness (Quirk, 2001; Blatz et al., 2004; Ueffing and Ney, 2007), human-likeness (Gamon et al., 2005) or continu- ous scores (Specia et al., 2009). As we also work with continuous scores, we are making an effort to combine previous feature acquisition sources, such as language modelling (Raybaud et al., 2009), language fluency checking (Parton et al., 2011), parsing (S´anchez-Martinez, 2011; Avramidis et al., 2011) and decoding statistics (Specia et al., 2009; Avramidis, 2011). The current submission combines such previous observations in a combinatory experi- mentation on feature sets, feature selection methods and Machine Learning (ML) algorithms. The structure of the submission is as follows: The approach is defined and the methods are described in section 2, including features acquisition, feature selection and learning. Section 3 includes informa- tion about the experiment setup whereas the results are discussed in Section 4. 2 Methods 2.1 Data and basic approach This contribution has been built based on the data released for the Quality Estimation task of the Workshop on Machine Translation (WMT) 2012 (Callison-Burch et al., 2012). The organizers pro- vided an English-to-Spanish development set and a test set of 1832 and 422 sentences respectively, de- rived from WMT09 and WMT10 datasets. For each source sentence of the development set, participants were offered one translation generated by a state-of- the-art phrase-based SMT system. The quality of each SMT translation was assessed by human evalu- ators, who provided a quality score in the range 1-5. Additionally, statistics and processing information from the execution of …

Content truncated to 500 words