Quality Estimation for Machine Translation Using the Joint Method of Evaluation Criteria and Statistical Modeling

Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 365–372, Sofia, Bulgaria, August 8-9, 2013 c⃝2013 Association for Computational Linguistics Quality Estimation for Machine Translation Using the Joint Method of Evaluation Criteria and Statistical Modeling Aaron Li-Feng Han hanlifengaaron@gmail.com Yi Lu mb25435@umac.mo Derek F. Wong derekfw@umac.mo Lidia S. Chao lidiasc@umac.mo Liangye He wutianshui0515@gmail.com Junwen Xing mb15470@umac.mo Natural Language Processing & Portuguese-Chinese Machine Translation Laboratory Department of Computer and Information Science University of Macau, Macau S.A.R. China Abstract This paper is to introduce our participation in the WMT13 shared tasks on Quality Estima- tion for machine translation without using ref- erence translations. We submitted the results for Task 1.1 (sentence-level quality estima- tion), Task 1.2 (system selection) and Task 2 (word-level quality estimation). In Task 1.1, we used an enhanced version of BLEU metric without using reference translations to evalu- ate the translation quality. In Task 1.2, we uti- lized a probability model Naïve Bayes (NB) as a classification algorithm with the features borrowed from the traditional evaluation met- rics. In Task 2, to take the contextual infor- mation into account, we employed a discrimi- native undirected probabilistic graphical mod- el Conditional random field (CRF), in addition to the NB algorithm. The training experiments on the past WMT corpora showed that the de- signed methods of this paper yielded promis- ing results especially the statistical models of CRF and NB. The official results show that our CRF model achieved the highest F-score 0.8297 in binary classification of Task 2. 1 Introduction Due to the fast development of Machine transla- tion, different automatic evaluation methods for the translation quality have been proposed in re- cent years. One of the categories is the lexical similarity based metric. This kind of metrics in- cludes the edit distance based method, such as WER (Su et al., 1992), Multi-reference WER (Nießen et al., 2000), PER (Tillmann et al., 1997), the works of (Akiba, et al., 2001), (Leusch et al., 2006) and (Wang and Manning, 2012); the precision based method, such as BLEU (Papineni et al., 2002), NIST (Doddington, 2002), and SIA (Liu and Gildea, 2006); recall based method, such as ROUGE (Lin and Hovy 2003); and the combination of precision and re- call, such as GTM (Turian et al., 2003), METE- OR (Lavie and Agarwal, 2007), BLANC (Lita et al., 2005), AMBER (Chen and Kuhn, 2011), PORT (Chen et al., 2012b), and LEPOR (Han et al., 2012). Another category is the using of linguistic fea- tures. This kind of metrics includes the syntactic similarity, such as the POS information used by TESLA (Dahlmeier et al., 2011), (Liu et al., 2010) and (Han et al., 2013), phrase information used by (Povlsen, et al., 1998) and (Echizen-ya and Araki, 2010), sentence structure used by (Owczarzak et al., 2007); the semantic similarity, such as textual entailment used by (Mirkin et al., 2009) and (Castillo and Estrella, 2012), Syno- nyms used by METEOR (Lavie and Agarwal, 2007), (Wong and Kit, 2012), (Chan and Ng, 2008); paraphrase used by (Snover …

Content truncated to 500 words