Document-Level Automatic MT Evaluation based on Discourse Representations

Proceedings of the Joint 5th Workshop on Statistical Machine Translation and MetricsMATR, pages 333–338, Uppsala, Sweden, 15-16 July 2010. c⃝2010 Association for Computational Linguistics Document-level Automatic MT Evaluation based on Discourse Representations Jes´us Gim´enez and Llu´ıs M`arquez TALP UPC Barcelona, Spain {jgimenez, lluism} @lsi.upc.edu Elisabet Comelles and Irene Castell´on Universitat de Barcelona Barcelona, Spain {elicomelles, icastellon} @ub.edu Victoria Arranz ELDA/ELRA Paris, France arranz@elda.org Abstract This paper describes the joint submission of Universitat Polit`ecnica de Catalunya and Uni- versitat de Barcelona to the Metrics MaTr 2010 evaluation challenge, in collaboration with ELDA/ELRA. Our work is aimed at widening the scope of current automatic evaluation measures from sentence to document level. Preliminary ex- periments, based on an extension of the metrics by Gim´enez and M`arquez (2009) operating over dis- course representations, are presented. 1 Introduction Current automatic similarity measures for Ma- chine Translation (MT) evaluation operate all, without exception, at the segment level. Trans- lations are analyzed on a segment-by-segment1 fashion, ignoring the text structure. Document and system scores are obtained using aggregate statistics over individual segments. This strategy presents the main disadvantage of ignoring cross- sentential/discursive phenomena. In this work we suggest widening the scope of evaluation methods. We have defined genuine document-level measures which are able to ex- ploit the structure of text to provide more informed evaluation scores. For that purpose we take advan- tage of two coincidental facts. First, test beds em- ployed in recent MT evaluation campaigns include a document structure grouping sentences related to the same event, story or topic (Przybocki et al., 2008; Przybocki et al., 2009; Callison-Burch et al., 2009). Second, we count on automatic linguistic processors which provide very detailed discourse- level representations of text (Curran et al., 2007). Discourse representations allow us to focus on relevant pieces of information, such as the agent 1A segment typically consists of one or two sentences. (who), location (where), time (when), and theme (what), which may be spread all over the text. Counting on a means of discerning the events, the individuals taking part in each of them, and their role, is crucial to determine the semantic equiva- lence between a reference document and a candi- date translation. Moreover, the discourse analysis of a document is not a mere concatenation of the analyses of its individual sentences. There are some phenom- ena which may go beyond the scope of a sen- tence and can only be explained within the con- text of the whole document. For instance, in a newspaper article, facts and entities are progres- sively added to the discourse and then referred to anaphorically later on. The following extract from the development set illustrates the impor- tance of such a phenomenon in the discourse anal- ysis: ‘Among the current or underlying crises in the Middle East, Rod Larsen mentioned the Arab- Israeli conflict and the Iranian nuclear portfolio, as well as the crisis between Lebanon and Syria. He stated: “All this leads us back to crucial val- ues and opinions, which render the situation …

Content truncated to 500 words