FBK-UPV-UEdin participation in the WMT14 Quality Estimation shared-task

Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 322–328, Baltimore, Maryland USA, June 26–27, 2014. c⃝2014 Association for Computational Linguistics FBK-UPV-UEdin participation in the WMT14 Quality Estimation shared-task Jos´e G. C. de Souza∗ University of Trento Fondazione Bruno Kessler Trento, Italy desouza@fbk.eu Jes´us Gonz´alez-Rubio∗ PRHLT Group U. Polit`ecnica de Val`encia Valencia, Spain jegonzalez@prhlt.upv.es Christian Buck∗ University of Edinburgh School of Informatics Edinburgh, Scotland, UK cbuck@lantis.de Marco Turchi, Matteo Negri Fondazione Bruno Kessler turchi,negri@fbk.eu Abstract This paper describes the joint submission of Fondazione Bruno Kessler, Universitat Polit`ecnica de Val`encia and University of Edinburgh to the Quality Estimation tasks of the Workshop on Statistical Machine Translation 2014. We present our submis- sions for Task 1.2, 1.3 and 2. Our systems ranked first for Task 1.2 and for the Binary and Level1 settings in Task 2. 1 Introduction Quality Estimation (QE) for Machine Translation (MT) is the task of evaluating the quality of the output of an MT system without reference transla- tions. Within the WMT 2014 QE Shared Task four evaluation tasks were proposed, covering both word and sentence level QE. In this work we de- scribe the Fondazione Bruno Kessler (FBK), Uni- versitat Polit`ecnica de Val`encia (UPV) and Uni- versity of Edinburgh (UEdin) approach and sys- tem setup for the shared task. We developed models for two sentence-level tasks: Task 1.2, scoring for post-editing effort, and Task 1.3, predicting post-editing time, and for all word-level variants of Task 2, binary and multiclass classification. As opposed to previous editions of the shared task, this year the partici- pants were not supplied with the MT system that was used to produce the translation. Furthermore no system-internal features were provided. Thus, while the trained models are tuned to detect the errors of a specific system the features have to be generated independently (black-box). 2 Sentence Level QE We submitted runs to two sentence-level tasks: Task 1.2 and Task 1.3. The first task aims at ∗Contributed equally to this work. predicting the Human mediated Translation Edit Rate (HTER) (Snover et al., 2006) between a sug- gestion generated by a machine translation sys- tem and its manually post-edited version. The data set contains 1,104 English-Spanish sentence pairs post-edited by one translator (896 for train- ing and 208 for test). The second task requires to predict the time, in milliseconds, that was re- quired to post edit a translation given by a ma- chine translation system. Participants are provided with 858 English-Spanish sentence pairs, source and suggestion, along with their respective post- edited sentence and post-editing time in seconds (650 data points for training and 208 for test). We participated in the scoring mode of both tasks. 2.1 Features For our sentence-level submissions we compute features using different resources that do not use the MT system internals. We use the same set of features for both Task 1.2 and 1.3. QuEst Black-box features (quest79). We ex- tract 79 black-box features that capture the com- plexity, fluency and adequacy aspects of the QE problem. These features …

Content truncated to 500 words