LIMSI Submission for the WMT‘13 Quality Estimation Task: an Experiment with N-Gram Posteriors

Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 398–404, Sofia, Bulgaria, August 8-9, 2013 c⃝2013 Association for Computational Linguistics LIMSI Submission for the WMT’13 Quality Estimation Task: an Experiment with n-gram Posteriors Anil Kumar Singh LIMSI Orsay, France anil@limsi.fr Guillaume Wisniewski Universit´e Paris Sud LIMSI Orsay, France wisniews@limsi.fr Franc¸ois Yvon Universit´e Paris Sud LIMSI Orsay, France yvon@limsi.fr Abstract This paper describes the machine learning algorithm and the features used by LIMSI for the Quality Estimation Shared Task. Our submission mainly aims at evaluating the usefulness for quality estimation of n- gram posterior probabilities that quantify the probability for a given n-gram to be part of the system output. 1 Introduction The dissemination of statistical machine transla- tion (SMT) systems in the professional translation industry is still limited by the lack of reliability of SMT outputs, the quality of which varies to a great extent. In this context, a critical piece of informa- tion would be for MT systems to assess their out- put translations with automatically derived quality measures. This problem is the focus of a shared task, the aim of which is to predict the quality of a translation without knowing any human ref- erence(s). To the best of our knowledge, all approaches so far have tackled quality estimation as a super- vised learning problem (He et al., 2010; Soricut and Echihabi, 2010; Specia et al., 2010; Specia, 2011). A wide variety of features have been pro- posed, most of which can be described as loosely ‘linguistic’ features that describe the source sen- tence, the target sentence and the association be- tween them (Callison-Burch et al., 2012). Sur- prisingly enough, information used by the decoder to choose the best translation in the search space, such as its internal scores, have hardly been con- sidered and never proved to be useful. Indeed, it is well-known that these scores are hard to interpret and to compare across hypotheses. Furthermore, mapping scores of a linear classifier (such as the scores estimated by MERT) into consistent prob- abilities is a difficult task (Platt, 2000; Lin et al., 2007). This work aims at assessing whether informa- tion extracted from the decoder search space can help to predict the quality of a translation. Rather than using directly the decoder score, we propose to consider a finer level of information, the n-gram posterior probabilities that quantifies the probabil- ity for a given n-gram to be part of the system output. These probabilities can be directly inter- preted as the confidence the system has for a given n-gram to be part of the translation. As they are directly derived from the number of hypotheses in the search space that contains this n-gram, these probabilities might be more reliable than the ones estimated from the decoder scores. We first quickly review, in Section 2, the n-gram posteriors introduced by (Gispert et al., 2013) and explain how they can be used in the QE task; we then describe, in Section 3 the different systems that have developed …

Content truncated to 500 words