Confidence estimation for translation prediction

Confidence Estimation for Translation Prediction Simona Gandrabur RALI, Universit´e de Montr´eal gandrabu@iro.umontreal.ca George Foster RALI, Universit´e de Montr´eal foster@iro.umontreal.ca Abstract The purpose of this work is to investigate the use of machine learning approaches for confi- dence estimation within a statistical machine translation application. Specifically, we at- tempt to learn probabilities of correctness for various model predictions, based on the native probabilites (i.e. the probabilites given by the original model) and on features of the current context. Our experiments were conducted us- ing three original translation models and two types of neural nets (single-layer and multi- layer perceptrons) for the confidence estima- tion task. 1 Introduction Most statistical models used in natural language appli- cations are capable in principle of generating probability estimates for their outputs. However, in practice, these estimates are often quite poor and are usually interpreted simply as scores that are monotonic with probabilities. There are many contexts where good estimates of true probabilities are desirable: • in a decision-theoretic setting, posterior probabili- ties are required in order to choose the lowest-cost output for a given input. • when a collection of different models is available for some problem, output probabilities provide a princi- pled and convenient way of combining them; and • when multiplying conditional probabilities to com- pute joint distributions, the accuracy of the result is crucially dependent on the stability of the con- ditional estimates across different contexts—this is important for applications like speech recognition and machine translation that perform searches over a large space of output sentences, represented as se- quences of words. Given a statistical model that produces a probabilistic score, a straightforward way of obtaining a true probabil- ity is to use the score as input to another model whose output is interpreted as the desired probability. The idea is that the second model can learn how to transform the base model’s score by observing its performance on new text, possibly in conjunction with other features. This ap- proach, which is known as confidence estimation (CE), is widely used in speech recognition (Guillevic et al., 2002; Moreno et al., 2001; Sanchis et al., 2003; Stolcke et al., 1997) but is virtually unknown in other areas of natural language progessing (NLP). 1 The alternatives to confidence estimation are tradi- tional smoothing techniques such as backing off to sim- pler models and cross validation, along with careful marginalization and scaling where applicable to obtain the desired posterior probabilities. There is some evi- dence (Wessel et al., 2001) that this approach can give results that are at least as good as those obtainable with an external CE model. However, CE as we present it here is not incompatible with traditional techniques, and has several practical advantages. First, it can easily incorpo- rate specialized features that are highly indicative of how well the base model will perform on a given input, but that may be of little use for the task of choosing the out- put. Since such features may be inconvenient to include in …

Content truncated to 500 words