Error Detection for Statistical Machine Translation Using Linguistic Features

Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 604–611, Uppsala, Sweden, 11-16 July 2010. c⃝2010 Association for Computational Linguistics Error Detection for Statistical Machine Translation Using Linguistic Features Deyi Xiong, Min Zhang, Haizhou Li Human Language Technology Institute for Infocomm Research 1 Fusionopolis Way, #21-01 Connexis, Singapore 138632. {dyxiong, mzhang, hli}@i2r.a-star.edu.sg Abstract Automatic error detection is desired in the post-processing to improve machine translation quality. The previous work is largely based on confidence estimation us- ing system-based features, such as word posterior probabilities calculated from N- best lists or word lattices. We propose to incorporate two groups of linguistic fea- tures, which convey information from out- side machine translation systems, into er- ror detection: lexical and syntactic fea- tures. We use a maximum entropy clas- sifier to predict translation errors by inte- grating word posterior probability feature and linguistic features. The experimen- tal results show that 1) linguistic features alone outperform word posterior probabil- ity based confidence estimation in error detection; and 2) linguistic features can further provide complementary informa- tion when combined with word confidence scores, which collectively reduce the clas- sification error rate by 18.52% and im- prove the F measure by 16.37%. 1 Introduction Translation hypotheses generated by a statistical machine translation (SMT) system always contain both correct parts (e.g. words, n-grams, phrases matched with reference translations) and incor- rect parts. Automatically distinguishing incorrect parts from correct parts is therefore very desir- able not only for post-editing and interactive ma- chine translation (Ueffing and Ney, 2007) but also for SMT itself: either by rescoring hypotheses in the N-best list using the probability of correct- ness calculated for each hypothesis (Zens and Ney, 2006) or by generating new hypotheses using N- best lists from one SMT system or multiple sys- tems (Akibay et al., 2004; Jayaraman and Lavie, 2005). In this paper we restrict the “parts” to words. That is, we detect errors at the word level for SMT. A common approach to SMT error detection at the word level is calculating the confidence at which a word is correct. The majority of word confidence estimation methods follows three steps: 1) Calculate features that express the correct- ness of words either based on SMT model (e.g. translation/language model) or based on SMT system output (e.g. N-best lists, word lattices) (Blatz et al., 2003; Ueffing and Ney, 2007). 2) Combine these features together with a clas- sification model such as multi-layer percep- tron (Blatz et al., 2003), Naive Bayes (Blatz et al., 2003; Sanchis et al., 2007), or log- linear model (Ueffing and Ney, 2007). 3) Divide words into two groups (correct trans- lations and errors) by using a classification threshold optimized on a development set. Sometimes the step 2) is not necessary if only one effective feature is used (Ueffing and Ney, 2007); and sometimes the step 2) and 3) can be merged into a single step if we directly output predicting results from binary classifiers instead of making thresholding decision. Various features …

Content truncated to 500 words