FBK-UEdin Participation to the WMT13 Quality Estimation Shared Task

Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 352–358, Sofia, Bulgaria, August 8-9, 2013 c⃝2013 Association for Computational Linguistics FBK-UEdin participation to the WMT13 Quality Estimation shared-task Jos´e G. C. de Souza FBK-irst University of Trento Trento, Italy desouza@fbk.eu Christian Buck School of Informatics University of Edinburgh Edinburgh, UK christian.buck@ed.ac.uk Marco Turchi, Matteo Negri FBK-irst Trento, Italy {turchi,negri}@fbk.eu Abstract In this paper we present the approach and system setup of the joint participation of Fondazione Bruno Kessler and University of Edinburgh in the WMT 2013 Quality Estimation shared-task. Our submissions were focused on tasks whose aim was pre- dicting sentence-level Human-mediated Translation Edit Rate and sentence-level post-editing time (Task 1.1 and 1.3, re- spectively). We designed features that are built on resources such as automatic word alignment, n-best candidate transla- tion lists, back-translations and word pos- terior probabilities. Our models consis- tently overcome the baselines for both tasks and performed particularly well for Task 1.3, ranking first among seven parti- cipants. 1 Introduction Quality Estimation (QE) for Machine Transla- tion (MT) is the task of evaluating the quality of the output of an MT system without relying on reference translations. The WMT 2013 QE Shared Task defined four different tasks covering both word and sentence level QE. In this work we describe the Fondazione Bruno Kessler (FBK) and University of Edinburgh approach and system setup of our participation to the shared task. We developed models for two sentence-level tasks: Task 1.1: Scoring and ranking for post-editing ef- fort, and Task 1.3: Predicting post-editing time. The first task aims at predicting the Human- mediated Translation Edit Rate (HTER) (Snover et al., 2006) between a suggestion generated by a machine translation system and its manually post-edited version. The data set contains 2,754 English-Spanish sentence pairs post-edited by one translator (2,254 for training and 500 for test). We participated only in the scoring mode of this task. The second task requires to predict the time, in seconds, that was required to post edit a transla- tion given by a machine translation system. Par- ticipants are provided with 1,087 English-Spanish sentence pairs, source and suggestion, along with their respective post-edited sentence and post- editing time in seconds (803 data points for train- ing and 284 for test). For both tasks we applied supervised learning methods and made use of information about word alignments, n-best diversity scores, word posterior probabilities, pseudo-references, and back trans- lation to train our models. In the remainder of this paper we describe the features designed for our participation (Section 2), the learning methods used to build our models (Section 3), the experi- ments that led to our submitted systems (Section 4), and we briefly conclude our experience in this evaluation task (Section 5). 2 Features 2.1 Word Alignment Information about word alignments is used to ex- tract quantitative (amount and distribution of the alignments) and qualitative features (importance of the aligned terms) under the assumption that features that explore what is aligned can bring im- provements to tasks …

Content truncated to 500 words