SHEF-Lite 2.0: Sparse Multi-task Gaussian Processes for Translation Quality Estimation

Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 307–312, Baltimore, Maryland USA, June 26–27, 2014. c⃝2014 Association for Computational Linguistics SHEF-Lite 2.0: Sparse Multi-task Gaussian Processes for Translation Quality Estimation Daniel Beck and Kashif Shah and Lucia Specia Department of Computer Science University of Sheffield Sheffield, United Kingdom {debeck1,kashif.shah,l.specia}@sheffield.ac.uk Abstract We describe our systems for the WMT14 Shared Task on Quality Estimation (sub- tasks 1.1, 1.2 and 1.3). Our submissions use the framework of Multi-task Gaus- sian Processes, where we combine multi- ple datasets in a multi-task setting. Due to the large size of our datasets we also ex- periment with Sparse Gaussian Processes, which aim to speed up training and predic- tion by providing sensible sparse approxi- mations. 1 Introduction The purpose of machine translation (MT) quality estimation (QE) is to provide a quality prediction for new, unseen machine translated texts, with- out relying on reference translations (Blatz et al., 2004; Specia et al., 2009; Bojar et al., 2013). A common use of quality predictions is the decision between post-editing a given machine translated sentence and translating its source from scratch, based on whether its post-editing effort is esti- mated to be lower than the effort of translating the source sentence. The WMT 2014 QE shared task defined a group of tasks related to QE. In this paper, we de- scribe our submissions for subtasks 1.1, 1.2 and 1.3. Our models are based on Gaussian Pro- cesses (GPs) (Rasmussen and Williams, 2006), a non-parametric kernelised probabilistic frame- work. We propose to combine multiple datasets to improve our QE models by applying GPs in a multi-task setting. Our hypothesis is that us- ing sensible multi-task learning settings gives im- provements over simply pooling all datasets to- gether. Task 1.1 focuses on predicting post-editing ef- fort for four language pairs: English-Spanish (en-es), Spanish-English (es-en), English-German (en-de), and German-English (de-en). Each con- tains a different number of source sentences and their human translations, as well as 2-3 versions of machine translations: by a statistical (SMT) system, a rule-based system (RBMT) system and, for en-es/de only, a hybrid system. Source sen- tences were extracted from tests sets of WMT13 and WMT12, and the translations were produced by top MT systems of each type and a human translator. Labels range from 1 to 3, with 1 in- dicating a perfect translation and 3, a low quality translation. The purpose of task 1.2 is to predict HTER scores (Human Translation Error Rate) (Snover et al., 2006) using a dataset composed of 896 English-Spanish sentences translated by a MT sys- tem and post-edited by a professional translator. Finally, task 1.3 aims at predicting post-editing time, using a subset of 650 sentences from the Task 1.2 dataset. For each task, participants can submit two types of results: scoring and ranking. For scoring, eval- uation is made in terms of Mean Absolute Error (MAE) and Root Mean Square Error (RMSE). For ranking, DeltaAvg and Spearman’s rank correla- tion were used as evaluation metrics. 2 Model Gaussian …

Content truncated to 500 words