SHEF-Lite: When Less is More for Translation Quality Estimation

Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 337–342, Sofia, Bulgaria, August 8-9, 2013 c⃝2013 Association for Computational Linguistics SHEF-Lite: When Less is More for Translation Quality Estimation Daniel Beck and Kashif Shah and Trevor Cohn and Lucia Specia Department of Computer Science University of Sheffield Sheffield, United Kingdom {debeck1,kashif.shah,t.cohn,l.specia}@sheffield.ac.uk Abstract We describe the results of our submissions to the WMT13 Shared Task on Quality Estimation (subtasks 1.1 and 1.3). Our submissions use the framework of Gaus- sian Processes to investigate lightweight approaches for this problem. We focus on two approaches, one based on feature se- lection and another based on active learn- ing. Using only 25 (out of 160) fea- tures, our model resulting from feature selection ranked 1st place in the scoring variant of subtask 1.1 and 3rd place in the ranking variant of the subtask, while the active learning model reached 2nd place in the scoring variant using only ∼25% of the available instances for train- ing. These results give evidence that Gaussian Processes achieve the state of the art performance as a modelling ap- proach for translation quality estimation, and that carefully selecting features and instances for the problem can further im- prove or at least maintain the same per- formance levels while making the problem less resource-intensive. 1 Introduction The purpose of machine translation (MT) quality estimation (QE) is to provide a quality prediction for new, unseen machine translated texts, with- out relying on reference translations (Blatz et al., 2004; Specia et al., 2009; Callison-burch et al., 2012). A common use of quality predictions is the decision between post-editing a given machine translated sentence and translating its source from scratch, based on whether its post-editing effort is estimated to be lower than the effort of translating the source sentence. The WMT13 QE shared task defined a group of tasks related to QE. In this paper, we present the submissions by the University of Sheffield team. Our models are based on Gaussian Pro- cesses (GP) (Rasmussen and Williams, 2006), a non-parametric probabilistic framework. We ex- plore the application of GP models in two con- texts: 1) improving the prediction performance by applying a feature selection step based on opti- mised hyperparameters and 2) reducing the dataset size (and therefore the annotation effort) by per- forming Active Learning (AL). We submitted en- tries for two of the four proposed tasks. Task 1.1 focused on predicting HTER scores (Human Translation Error Rate) (Snover et al., 2006) using a dataset composed of 2254 English- Spanish news sentences translated by Moses (Koehn et al., 2007) and post-edited by a profes- sional translator. The evaluation used a blind test set, measuring MAE (Mean Absolute Error) and RMSE (Root Mean Square Error), in the case of the scoring variant, and DeltaAvg and Spearman’s rank correlation in the case of the ranking vari- ant. Our submissions reached 1st (feature selec- tion) and 2nd (active learning) places in the scor- ing variant, the task the models were optimised for, and outperformed the …

Content truncated to 500 words