Word- and Sentence-Level Confidence Measures for Machine Translation

Proceedings of the 13th Annual Conference of the EAMT, pages 104–111, Barcelona, May 2009 Word- and Sentence-level Confidence Measures for Machine Translation Sylvain Raybaud, Caroline Lavecchia, David Langlois, Kamel Sma¨ıli PAROLE team, LORIA Campus Scientifique BP 239 54506 Vandoeuvre-l`es-Nancy FRANCE {sylvain.raybaud,caroline.lavecchia,david.langlois,kamel.smaili}@loria.fr Abstract A machine translated sentence is seldom completely correct. Confidence measures are designed to detect incorrect words, phrases or sentences, or to provide an esti- mation of the probability of correctness. In this article we describe several word- and sentence-level confidence measures rely- ing on different features: mutual informa- tion between words, n-gram and backward n-gram language models, and linguistic features. We also try different combination of these measures. Their accuracy is eval- uated on a classification task. We achieve 17% error-rate (0.84 f-measure) on word- level and 31% error-rate (0.71 f-measure) on sentence-level. 1 Introduction Statistical techniques have been widely used and remarkably successful in automatic speech recog- nition, machine translation and in natural language processing over the last two decades. This success is due to the fact that this approach is language independent and requires no prior knowledge, only large enough text corpora to estimate probability densities on. However statistical methods suffer from an intrinsic drawback: they only produce the result which is most likely given training and input data. It is easy to see that this will sometimes not be optimal with regard to human expectations. It is therefore important to be able to automatically evaluate the quality of the result: this can be han- dled by the different confidence measures (CMs) which have been proposed for machine translation. c⃝ 2009 European Association for Machine Translation. This paper extends and improve the work pre- sented in (Raybaud et al., 2009): we introduce new CMs to assess the reliability of translation results. The proposed CMs take advantage of the con- stituents of a translated sentence: n-grams, word triggers, and also word features. We also combine the scores given by the different measures in order to produce a new one, hopefully more powerful, and the scores given to the different words in order to estimate the whole sentence’s reliability. 1.1 A brief overview of statistical machine translation In this framework the translation process is essen- tially the search for the most probable sentence in the target language given a sentence in the source language; let s = s1,..,sI be the source sentence (to be translated) and ˆt = t1,..,tJ be the sentence gen- erated in the target language by the system: ˆt = argmax t P(t|s) (1) which is equivalent (using the Bayes rule) to: ˆt = argmax t P(t)P(s|t) (2) In Equation 2, P(t) is estimated from a language model and is supposed to estimate the correctness of the sentence (“is it a good sentence in the tar- get language ?”), and P(s|t) is computed from a translation model and is supposed to reflect the accuracy of the translation (“does the generated sentence carry exactly the same information than the source sentence ?”). The language model is …

Content truncated to 500 words