LIMSI Submission for WMT’14 QE Task

Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 348–354, Baltimore, Maryland USA, June 26–27, 2014. c⃝2014 Association for Computational Linguistics LIMSI Submission for WMT’14 QE Task Guillaume Wisniewski and Nicolas P´echeux and Alexandre Allauzen and Franc¸ois Yvon Universit´e Paris Sud and LIMSI-CNRS 91 403 ORSAY CEDEX, France {wisniews, pecheux, allauzen, yvon}@limsi.fr Abstract This paper describes LIMSI participation to the WMT’14 Shared Task on Qual- ity Estimation; we took part to the word- level quality estimation task for English to Spanish translations. Our system re- lies on a random forest classifier, an en- semble method that has been shown to be very competitive for this kind of task, when only a few dense and continuous fea- tures are used. Notably, only 16 features are used in our experiments. These fea- tures describe, on the one hand, the qual- ity of the association between the source sentence and each target word and, on the other hand, the fluency of the hypothe- sis. Since the evaluation criterion is the f1 measure, a specific tuning strategy is proposed to select the optimal values for the hyper-parameters. Overall, our system achieves a 0.67 f1 score on a randomly ex- tracted test set. 1 Introduction This paper describes LIMSI submission to the WMT’14 Shared Task on Quality Estimation. We participated in the word-level quality estimation task (Task 2) for the English to Spanish direction. This task consists in predicting, for each word in a translation hypothesis, whether this word should be post-edited or should rather be kept unchanged. Predicting translation quality at the word level raises several interesting challenges. First, this is a (relatively) new task and the best way to for- mulate and evaluate it has still to be established. Second, as most works on quality estimation have only considered prediction at the sentence level, it is not clear yet which features are really effective to predict quality at the word and a set of base- line features has still to be found. Finally, sev- eral characteristic of the task (the limited number of training examples, the unbalanced classes, etc.) makes the use of ‘traditional’ machine learning al- gorithms difficult. This papers describes how we addressed this different issues for our participation to the WMT’14 Shared Task. The rest of this paper is organized as follows. Section 2 gives an overview of the shared task data that will justify some of the design decisions we made. Section 3 describes the different features we have considered and Section 4, the learning methods used to estimate the classifiers parame- ters. Finally the results of our models are pre- sented and analyzed in Section 5. 2 World-Level Quality Estimation WMT’14 shared task on quality estimation num- ber 2 consists in predicting, for each word of a translation hypothesis, whether this word should be post-edited (denoted by the BAD label) or should be kept unchanged (denoted by the OK la- bel). The shared task organizers provide a bilin- gual dataset from English to Spanish1 made of translations …

Content truncated to 500 words