An Approach Using Style Classification Features for Quality Estimation

Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 429–434, Sofia, Bulgaria, August 8-9, 2013 c⃝2013 Association for Computational Linguistics An approach using style classification features for Quality Estimation Erwan Moreau CNGL and Computational Linguistics Group Centre for Computing and Language Studies School of Computer Science and Statistics Trinity College Dublin Dublin 2, Ireland moreaue@cs.tcd.ie Raphael Rubino NCLT Dublin City University Dublin 9, Ireland rrubino@computing.dcu.ie Abstract In this paper we describe our participation to the WMT13 Shared Task on Quality Es- timation. The main originality of our ap- proach is to include features originally de- signed to classify text according to some author’s style. This implies the use of ref- erence categories, which are meant to rep- resent the quality of the MT output. Preamble This paper describes the approach followed in the two systems that we submitted to subtask 1.3 of the WMT13 Shared Task on Quality Estimation, identified as TCD-DCU-CNGL 1-3 SVM1 and TCD-DCU-CNGL 1-3 SVM2. This approach was also used by the first author in his submissions to subtask 1.1, identified as TCD-CNGL OPEN and TCD-CNGL RESTRICTED1. In the remain- ing of this paper we focus on subtask 1.3, but there is very little difference in the application of the ap- proach to task 1.1. 1 Introduction Quality Estimation (QE) aims to provide a quality indicator for machine translated sentences. There are many cases where such an indicator would be useful in a translation process: to compare differ- ent Machine Translation (MT) models on a given set of sentences, to tune automatically the param- eters of a MT model, to select the bad sentences for human translation or post-editing, to select the good sentences for immediate publication and try to apply automatic post-editing to the others, or simply to provide users who are not fluent in the source language information about the fluency of 1The second author’s submission to subtask 1.1 is inde- pendent from this approach and is described in a different paper in this volume. the translated text they are reading. As long as ma- chine translated text cannot be of reasonably con- sistent quality, QE is helpful in indicating linguis- tic quality variability.2 After focusing on automatic prediction of ad- hoc quality scores (as estimated by professional annotators) in the previous edition (Callison- Burch et al., 2012), the WMT Shared Task on Quality Estimation 2013 proposes several variants of the task. We participated in task 1.1 which aims to predict HTER scores (edit distance between the MT output and its manually post-edited version), and in task 1.3 which aims to predict the expected time needed to post-edit the MT output. The originality of our participation lies in the fact that we intended to test “style classification” features for the task of QE: the idea is to select a set of n-grams which are particularly representa- tive of a given level of quality. In practice we use only two levels which simply represent low and high quality. We explore various ways to build these …

Content truncated to 500 words