MT Tuning on RED: A Dependency-Based Evaluation Metric

Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 428–433, Lisboa, Portugal, 17-18 September 2015. c⃝2015 Association for Computational Linguistics. MT Tuning on RED: A Dependency-Based Evaluation Metric Liangyou Li∗ Hui Yu† Qun Liu∗† ∗ ADAPT Centre, School of Computing Dublin City University, Ireland † Key Laboratory of Intelligent Information Processing Institute of Computing Technology Chinese Academy of Sciences, China {liangyouli,qliu}@computing.dcu.ie yuhui@ict.ac.cn Abstract In this paper, we describe our submis- sion to WMT 2015 Tuning Task. We integrate a dependency-based MT evalu- ation metric, RED, to Moses and com- pare it with BLEU and METEOR in con- junction with two tuning methods: MERT and MIRA. Experiments are conducted us- ing hierarchical phrase-based models on Czech–English and English–Czech tasks. Our results show that MIRA performs bet- ter than MERT in most cases. Using RED performs similarly to METEOR when tun- ing is performed using MIRA. We submit our system tuned by MIRA towards RED to WMT 2015. In human evaluations, we achieve the 1st rank in all 7 systems on the English–Czech task and 6/9 on the Czech– English task. 1 Introduction Statistical Machine Translation (SMT) is modeled as a weighted combination of several features. Tuning in SMT refers to learning a set of opti- mized weights, which minimize a defined trans- lation error on a tuning set. Typically, the er- ror is measured by an automatic evaluation met- ric. Thanks to its simplicity and language indepen- dence, BLEU (Papineni et al., 2002) has served as the optimization objective since the 2000s. Although various lexical metrics, such as TER (Snover et al., 2006) and METEOR (Lavie and Denkowski, 2009) etc., have been proposed, none of them can truly replace BLEU in a phrase-based system (Cer et al., 2010). However, BLEU has no proficiency to deal with synonyms, paraphrases, and syntactic equivalent etc. (Callison-Burch et al., 2006). In addition, as a lexical and n-gram-based metric, BLEU may be not suitable for optimization in a syntax-based model. In this paper, we integrate a reference dependency-based MT evaluation metric, RED1 (Yu et al., 2014), into the hierarchical phrase- based model (Chiang, 2005) in Moses (Koehn et al., 2007). In doing so, we explore whether a syntax-based translation system will perform better when it is optimized towards a syntax- based evaluation criteria. We compare RED with two other evaluation metrics, BLEU and ME- TEOR (Section 2). Two tuning algorithms are used (Section 3). They are MERT (Och, 2003), MIRA (Cherry and Foster, 2012). Experiments are conducted on Czech–English and English–Czech translation (Section 4). 2 Evaluation Metrics An evaluation metric, which has a higher correla- tion with human judgments, may be used to train a better system. In this paper, we compare three metrics: BLEU, METEOR, and RED. 2.1 BLEU BLEU is the most widely used metric in SMT. It is lexical-based and language-independent. BLEU scores a hypothesis by combining n-gram pre- cisions over reference translations with a length penalty. A n-gram precision pn is calculated separately for different n-gram lengths. BLEU combines these precisions …

Content truncated to 500 words