Goodness: A Method for Measuring Machine Translation Confidence

Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics, pages 211–219, Portland, Oregon, June 19-24, 2011. c⃝2011 Association for Computational Linguistics Goodness: A Method for Measuring Machine Translation Confidence Nguyen Bach∗ Language Technologies Institute Carnegie Mellon University Pittsburgh, PA 15213, USA nbach@cs.cmu.edu Fei Huang and Yaser Al-Onaizan IBM T.J. Watson Research Center 1101 Kitchawan Rd Yorktown Heights, NY 10567, USA {huangfe, onaizan}@us.ibm.com Abstract State-of-the-art statistical machine translation (MT) systems have made significant progress towards producing user-acceptable translation output. However, there is still no efficient way for MT systems to inform users which words are likely translated correctly and how confident it is about the whole sentence. We propose a novel framework to predict word- level and sentence-level MT errors with a large number of novel features. Experimental re- sults show that the MT error prediction accu- racy is increased from 69.1 to 72.2 in F-score. The Pearson correlation between the proposed confidence measure and the human-targeted translation edit rate (HTER) is 0.6. Improve- ments between 0.4 and 0.9 TER reduction are obtained with the n-best list reranking task us- ing the proposed confidence measure. Also, we present a visualization prototype of MT er- rors at the word and sentence levels with the objective to improve post-editor productivity. 1 Introduction State-of-the-art Machine Translation (MT) systems are making progress to generate more usable translation outputs. In particular, statistical machine translation systems (Koehn et al., 2007; Bach et al., 2007; Shen et al., 2008) have advanced to a state that the transla- tion quality for certain language pairs (e.g. Spanish- English, French-English, Iraqi-English) in certain do- mains (e.g. broadcasting news, force-protection, travel) is acceptable to users. However, a remaining open question is how to pre- dict confidence scores for machine translated words and sentences. An MT system typically returns the best translation candidate from its search space, but still has no reliable way to inform users which word is likely to be correctly translated and how confident it is about the whole sentence. Such information is vital ∗ Work done during an internship at IBM T.J. Watson Research Center to realize the utility of machine translation in many ar- eas. For example, a post-editor would like to quickly identify which sentences might be incorrectly trans- lated and in need of correction. Other areas, such as cross-lingual question-answering, information extrac- tion and retrieval, can also benefit from the confidence scores of MT output. Finally, even MT systems can leverage such information to do n-best list reranking, discriminative phrase table and rule filtering, and con- straint decoding (Hildebrand and Vogel, 2008). Numerous attempts have been made to tackle the confidence estimation problem. The work of Blatz et al. (2004) is perhaps the best known study of sentence and word level features and their impact on transla- tion error prediction. Along this line of research, im- provements can be obtained by incorporating more fea- tures as shown in (Quirk, 2004; Sanchis et al., 2007; Raybaud et al., 2009; Specia et al., 2009). Sori- …

Content truncated to 500 words