Word-Level Confidence Estimation for Machine Translation using Phrase-Based Translation Models

Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing (HLT/EMNLP), pages 763–770, Vancouver, October 2005. c⃝2005 Association for Computational Linguistics Word-Level Confidence Estimation for Machine Translation using Phrase-Based Translation Models Nicola Ueffing and Hermann Ney Lehrstuhl f¨ur Informatik VI, Computer Science Department RWTH Aachen University D-52056 Aachen, Germany {ueffing,ney}@informatik.rwth-aachen.de Abstract Confidence measures for machine transla- tion is a method for labeling each word in an automatically generated translation as correct or incorrect. In this paper, we will present a new approach to confidence es- timation which has the advantage that it does not rely on system output such as N- best lists or word graphs as many other confidence measures do. It is, thus, appli- cable to any kind of machine translation system. Experimental evaluation has been per- formed on translation of technical manu- als in three different language pairs. Re- sults will be presented for different ma- chine translation systems to show that the new approach is independent of the under- lying machine translation system which generated the translations. To the best of our knowledge, the performance of the new confidence measure is better than that of any existing confidence measure. 1 Introduction The work presented in this paper deals with con- fidence estimation for machine translation (MT). Since sentences produced by a machine translation system are often incorrect but may contain correct parts, a method for identifying those correct parts and finding possible errors is desirable. For this pur- pose, each word in the generated target sentence is assigned a value expressing the confidence that it is correct. Confidence measures have been extensively stud- ied for speech recognition, but are not well known in other areas. Only recently have researchers started to investigate confidence measures for ma- chine translation (Blatz et al., 2004; Gandrabur and Foster, 2003; Quirk, 2004; Ueffing et al., 2003). We apply word confidence measures in MT as fol- lows: For a given translation generated by a machine translation system, we determine a confidence value for each word and compare it to a threshold. All words whose confidence is above this threshold are tagged as correct and all others are tagged as incor- rect translations. The threshold is optimized on a distinct development set beforehand. Possible applications for confidence measures in- clude • post-editing, where words with low confidence could be marked as potential errors, • improving translation prediction accuracy in trans-type-style interactive machine transla- tion (Gandrabur and Foster, 2003; Ueffing and Ney, 2005), • combining output from different machine translation systems: hypotheses with low confi- dence can be discarded before selecting one of the system translations (Akiba et al., 2004), or the word confidence scores can be used for gen- erating new hypotheses from the output of dif- ferent systems (Jayaraman and Lavie, 2005), or the sentence confidence value can be employed for re-ranking (Blatz et al., 2003). In this paper, we will present several approaches to word-level confidence estimation and develop a new phrase-based confidence measure which …

Content truncated to 500 words