Morpheme- and POS-based IBM1 and language model scores for translation quality estimation

Proceedings of the 7th Workshop on Statistical Machine Translation, pages 133–137, Montr´eal, Canada, June 7-8, 2012. c⃝2012 Association for Computational Linguistics Morpheme- and POS-based IBM1 scores and language model scores for translation quality estimation Maja Popovi´c German Research Center for Artificial Intelligence (DFKI) Language Technology (LT), Berlin, Germany maja.popovic@dfki.de Abstract We present a method we used for the quality estimation shared task of WMT 2012 involving IBM1 and language model scores calculated on morphemes and POS tags. The IBM1 scores calculated on morphemes and POS-4grams of the source sentence and obtained translation output are shown to be competitive with the classic evaluation metrics for ranking of trans- lation systems. Since these scores do not re- quire any reference translations, they can be used as features for the quality estimation task presenting a connection between the source language and the obtained target language. In addition, target language model scores of mor- phemes and POS tags are investigated as esti- mates for the obtained target language quality. 1 Introduction Automatic quality estimation is a topic of increas- ing interest in machine translation. Different from evaluation task, quality estimation does not rely on any reference translations – it relies only on infor- mation about the input source text, obtained target language text, and translation process. Being a new topic, it still does not have well established base- lines, datasets or standard evaluation metrics. The usual approach is to use a set of features which are used to train a classifier in order to assign a predic- tion score to each sentence. In this work, we propose a set of features based on the morphological and syntactic properties of in- volved languages thus abstracting away from word surface particularities (such as vocabulary and do- main). This approach is shown to be very useful for evaluation task (Popovi´c, 2011; Popovi´c et al., 2011; Callison-Burch et al., 2011). The features investi- gated in this work are based on the language model (LM) scores and on the IBM1 lexicon scores (Brown et al., 1993). The inclusion of IBM1 scores in translation sys- tems has shown experimentally to improve transla- tion quality (Och et al., 2003). They also have been used for confidence estimation for machine transla- tion (Blatz et al., 2003). The IBM1 scores calcu- lated on morphemes and POS-4grams are shown to be competitive with the classic evaluation metrics based on comparison with given reference transla- tions (Popovi´c et al., 2011; Callison-Burch et al., 2011). To the best of our knowledge, these scores have not yet been used for translation quality esti- mation. The LM scores of words and POS tags are used for quality estimation in previous work (Spe- cia et al., 2009), and in our work we investigate the scores calculated on morphemes and POS tags. At this point, only preliminary experiments have been carried out in order to determine if the pro- posed features are promising at all. We did not use any classifier, we used the obtained scores to rank the sentences …

Content truncated to 500 words