DCU-Symantec Submission for the WMT 2012 Quality Estimation Task

Proceedings of the 7th Workshop on Statistical Machine Translation, pages 138–144, Montr´eal, Canada, June 7-8, 2012. c⃝2012 Association for Computational Linguistics DCU-Symantec Submission for the WMT 2012 Quality Estimation Task Raphael Rubino†‡, Jennifer Foster†, Joachim Wagner†, Johann Roturier‡, Rasul Samad Zadeh Kaljahi†‡, Fred Hollowood‡ †Dublin City University, ‡Symantec, Ireland †firstname.lastname@computing.dcu.ie ‡firstname lastname@symantec.com Abstract This paper describes the features and the ma- chine learning methods used by Dublin City University (DCU) and SYMANTEC for the WMT 2012 quality estimation task. Two sets of features are proposed: one constrained, i.e. respecting the data limitation suggested by the workshop organisers, and one unconstrained, i.e. using data or tools trained on data that was not provided by the workshop organisers. In total, more than 300 features were extracted and used to train classifiers in order to predict the translation quality of unseen data. In this paper, we focus on a subset of our feature set that we consider to be relatively novel: fea- tures based on a topic model built using the Latent Dirichlet Allocation approach, and fea- tures based on source and target language syn- tax extracted using part-of-speech (POS) tag- gers and parsers. We evaluate nine feature combinations using four classification-based and four regression-based machine learning techniques. 1 Introduction For the first time, the WMT organisers this year pro- pose a Quality Estimation (QE) shared task, which is divided into two sub-tasks: scoring and ranking automatic translations. The aim of this workshop is to define useful sets of features and machine learn- ing techniques in order to predict the quality of a machine translation (MT) output T (Spanish) given a source segment S (English). Quality is measured using a 5-point likert scale which is based on post- editing effort, following the scoring scheme: 1. The MT output is incomprehensible 2. About 50-70% of the MT output needs to be edited 3. About 25-50% of the MT output needs to be edited 4. About 10-25% of the MT output needs to be edited 5. The MT output is perfectly clear and intelligi- ble The final score is a combination of the scores as- signed by three evaluators. The use of a 5-point scale makes the scoring task more difficult than a binary classification task where a translation is considered to be either good or bad. However, if the task is successfully carried out, the score produced is more useful. Dublin City University and Symantec jointly ad- dress the scoring task. For each pair (S, T) of source segment S and machine translation T, we train three classifiers and one classifier combination using the training data provided by the organisers to predict 5-point Likert scores. In this paper, we present the classification results on the test set along with addi- tional results obtained using regression techniques. We evaluate the usefulness of two new sets of fea- tures: 1. topic-based features using Latent Dirichlet Al- location (LDA (Blei et al., 2003)), 2. syntax-based features using POS taggers and parsers (Wagner et al., 2009) The remainder …

Content truncated to 500 words