LORIA System for the WMT15 Quality Estimation Shared Task

Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 323–329, Lisboa, Portugal, 17-18 September 2015. c⃝2015 Association for Computational Linguistics. LORIA System for the WMT15 Quality Estimation Shared Task Langlois David SMarT Group, LORIA Inria, Villers-l`es-Nancy, F-54600, France Universit de Lorraine, LORIA, UMR 7503, Villers-l`es-Nancy, F-54600, France CNRS, LORIA, UMR 7503, Villers-l`es-Nancy, F-54600, France david.langlois@loria.fr Abstract We describe our system for WMT2015 Shared Task on Quality Estimation, task 1, sentence-level prediction of post-edition effort. We use baseline features, Latent Semantic Indexing based features and fea- tures based on pseudo-references. SVM algorithm allows to estimate the linear re- gression between the features vectors and the HTER score. We use a selection al- gorithm in order to put aside needless fea- tures. Our best system leads to a perfor- mance in terms of Mean Absolute Error equal to 13.34 on official test while the official baseline system leads to a perfor- mance equal to 14.82. 1 Introduction This paper describes the LORIA submission to the WMT’15 Shared Task on Quality Estimation. We participated to the Task 1. This task consists in predicting the edition effort needed to correct a translated sentence. The organizers provide En- glish sentences automatically translated into Span- ish, and the corresponding post-edited sentences. The edition effort is measured by edit-distance rate (HTER (Snover et al., 2006)) between the trans- lated sentence and its post-edited version. Classically, our system extracts numerical fea- tures from sentences and applies a machine learn- ing approach between numeric vectors and HTER scores. As last year, no information is given about the Machine Translation (MT) system used to build data. Therefore, it is only possible to use blackbox features, or to use other MT systems whom output is compared to the evaluated target sentence. Our submission deals with the both kinds of fea- tures. First, we use a Latent Semantic Analysis ap- proach to measure the lexical similarity between a source and a target sentence. To our knowledge, this approach has never been used in the scope of Quality Estimation. Second, we use the output of 3 online MT systems, and we extract information about the intersection between the evaluated target sentence and the 3 translated sentences by online systems. This intersection is measured in terms of shared 1,2,3,4-grams. The paper is structured as follows. Section 2 give details about experimental protocol and used data. We describe the features we use in Section 3. Then, we give results (Section 4) and we conclude. 2 Experimental protocol and used corpus In this section, we describe how we obtain re- sults starting from training, development and test corpus. The training and development corpus are composed of a set of triplets. Each triplet is made up of a source sentence, its automatic translation, and a score representing the translation quality. For our experiments, we use the corpora the organizers provide. The source language is En- glish, the target language is Spanish. For each source sentence s, a machine translation system (unknown to the participants) gives a …

Content truncated to 500 words