Using Discourse Structure Improves Machine Translation Evaluation

Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, pages 687–698, Baltimore, Maryland, USA, June 23-25 2014. c⃝2014 Association for Computational Linguistics Using Discourse Structure Improves Machine Translation Evaluation Francisco Guzm´an Shafiq Joty Llu´ıs M`arquez and Preslav Nakov ALT Research Group Qatar Computing Research Institute — Qatar Foundation {fguzman,sjoty,lmarquez,pnakov}@qf.org.qa Abstract We present experiments in using dis- course structure for improving machine translation evaluation. We first design two discourse-aware similarity measures, which use all-subtree kernels to compare discourse parse trees in accordance with the Rhetorical Structure Theory. Then, we show that these measures can help improve a number of existing machine translation evaluation metrics both at the segment- and at the system-level. Rather than proposing a single new metric, we show that discourse information is com- plementary to the state-of-the-art evalu- ation metrics, and thus should be taken into account in the development of future richer evaluation metrics. 1 Introduction From its foundations, Statistical Machine Transla- tion (SMT) had two defining characteristics: first, translation was modeled as a generative process at the sentence-level. Second, it was purely statisti- cal over words or word sequences and made lit- tle to no use of linguistic information. Although modern SMT systems have switched to a discrim- inative log-linear framework, which allows for ad- ditional sources as features, it is generally hard to incorporate dependencies beyond a small window of adjacent words, thus making it difficult to use linguistically-rich models. Recently, there have been two promising re- search directions for improving SMT and its eval- uation: (a) by using more structured linguistic information, such as syntax (Galley et al., 2004; Quirk et al., 2005), hierarchical structures (Chi- ang, 2005), and semantic roles (Wu and Fung, 2009; Lo et al., 2012), and (b) by going beyond the sentence-level, e.g., translating at the docu- ment level (Hardmeier et al., 2012). Going beyond the sentence-level is important since sentences rarely stand on their own in a well-written text. Rather, each sentence follows smoothly from the ones before it, and leads into the ones that come afterwards. The logical rela- tionship between sentences carries important in- formation that allows the text to express a meaning as a whole beyond the sum of its separate parts. Note that sentences can be made of several clauses, which in turn can be interrelated through the same logical relations. Thus, in a coherent text, discourse units (sentences or clauses) are logically connected: the meaning of a unit relates to that of the previous and the following units. Discourse analysis seeks to uncover this coher- ence structure underneath the text. Several formal theories of discourse have been proposed to de- scribe the coherence structure (Mann and Thomp- son, 1988; Asher and Lascarides, 2003; Webber, 2004). For example, the Rhetorical Structure The- ory (Mann and Thompson, 1988), or RST, repre- sents text by labeled hierarchical structures called Discourse Trees (DTs), which can incorporate sev- eral layers of other linguistic information, e.g., syntax, predicate-argument structure, etc. Modeling discourse brings together the above research …

Content truncated to 500 words