DiscoTK: Using Discourse Structure for Machine Translation Evaluation

Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 402–408, Baltimore, Maryland USA, June 26–27, 2014. c⃝2014 Association for Computational Linguistics DiscoTK: Using Discourse Structure for Machine Translation Evaluation Shafiq Joty Francisco Guzm´an Llu´ıs M`arquez and Preslav Nakov ALT Research Group Qatar Computing Research Institute — Qatar Foundation {sjoty,fguzman,lmarquez,pnakov}@qf.org.qa Abstract We present novel automatic metrics for machine translation evaluation that use discourse structure and convolution ker- nels to compare the discourse tree of an automatic translation with that of the hu- man reference. We experiment with five transformations and augmentations of a base discourse tree representation based on the rhetorical structure theory, and we combine the kernel scores for each of them into a single score. Finally, we add other metrics from the ASIYA MT evaluation toolkit, and we tune the weights of the combination on actual human judgments. Experiments on the WMT12 and WMT13 metrics shared task datasets show corre- lation with human judgments that outper- forms what the best systems that partici- pated in these years achieved, both at the segment and at the system level. 1 Introduction The rapid development of statistical machine translation (SMT) that we have seen in recent years would not have been possible without au- tomatic metrics for measuring SMT quality. In particular, the development of BLEU (Papineni et al., 2002) revolutionized the SMT field, al- lowing not only to compare two systems in a way that strongly correlates with human judg- ments, but it also enabled the rise of discrimina- tive log-linear models, which use optimizers such as MERT (Och, 2003), and later MIRA (Watanabe et al., 2007; Chiang et al., 2008) and PRO (Hop- kins and May, 2011), to optimize BLEU, or an ap- proximation thereof, directly. While over the years other strong metrics such as TER (Snover et al., 2006) and Meteor (Lavie and Denkowski, 2009) have emerged, BLEU remains the de-facto stan- dard, despite its simplicity. Recently, there has been steady increase in BLEU scores for well-resourced language pairs such as Spanish-English and Arabic-English. However, it was also observed that BLEU-like n- gram matching metrics are unreliable for high- quality translation output (Doddington, 2002; Lavie and Agarwal, 2007). In fact, researchers al- ready worry that BLEU will soon be unable to dis- tinguish automatic from human translations.1 This is a problem for most present-day metrics, which cannot tell apart raw machine translation output from a fully fluent professionally post-edited ver- sion thereof (Denkowski and Lavie, 2012). Another concern is that BLEU-like n-gram matching metrics tend to favor phrase-based SMT systems over rule-based systems and other SMT paradigms. In particular, they are unable to cap- ture the syntactic and semantic structure of sen- tences, and are thus insensitive to improvement in these aspects. Furthermore, it has been shown that lexical similarity is both insufficient and not strictly necessary for two sentences to convey the same meaning (Culy and Riehemann, 2003; Coughlin, 2003; Callison-Burch et al., 2006). The above issues have motivated a large amount of work dedicated to design …

Content truncated to 500 words