On the Robustness of Syntactic and Semantic Features for Automatic MT Evaluation

Proceedings of the Fourth Workshop on Statistical Machine Translation , pages 250–258, Athens, Greece, 30 March – 31 March 2009. c⃝2009 Association for Computational Linguistics On the Robustness of Syntactic and Semantic Features for Automatic MT Evaluation Jes´us Gim´enez and Llu´ıs M`arquez TALP Research Center, LSI Department Universitat Polit`ecnica de Catalunya Jordi Girona Salgado 1–3, E-08034, Barcelona {jgimenez,lluism}@lsi.upc.edu Abstract Linguistic metrics based on syntactic and semantic information have proven very effective for Automatic MT Evaluation. However, no results have been presented so far on their performance when applied to heavily ill-formed low quality transla- tions. In order to glean some light into this issue, in this work we present an empirical study on the behavior of a heterogeneous set of metrics based on linguistic analysis in the paradigmatic case of speech transla- tion between non-related languages. Cor- roborating previous findings, we have ver- ified that metrics based on deep linguis- tic analysis exhibit a very robust and sta- ble behavior at the system level. How- ever, these metrics suffer a significant de- crease at the sentence level. This is in many cases attributable to a loss of recall, due to parsing errors or to a lack of parsing at all, which may be partially ameliorated by backing off to lexical similarity. 1 Introduction Recently, there is a growing interest in the devel- opment of automatic evaluation metrics which ex- ploit linguistic knowledge at the syntactic and se- mantic levels. For instance, we may find metrics which compute similarities over shallow syntac- tic structures/sequences (Gim´enez and M`arquez, 2007; Popovic and Ney, 2007), constituency trees (Liu and Gildea, 2005) and dependency trees (Liu and Gildea, 2005; Amig´o et al., 2006; Mehay and Brew, 2007; Owczarzak et al., 2007). We may also find metrics operating over shallow semantic structures, such as named entities and se- mantic roles (Gim´enez and M`arquez, 2007). Linguistic metrics have been proven to produce more reliable system rankings than metrics limit- ing their scope to the lexical dimension, in partic- ular when applied to test beds with a rich system typology, i.e., test beds in which there are auto- matic outputs produced by systems based on dif- ferent paradigms, e.g., statistical, rule-based and human-aided (Gim´enez and M`arquez, 2007). The reason is that they are able to capture deep MT quality distinctions which occur beyond the shal- low level of lexical similarities. However, these metrics have the limitation of relying on automatic linguistic processors, tools which are not equally available for all languages and whose performance may vary depending on the type of analysis conducted and the applica- tion domain. Thus, it could be argued that lin- guistic metrics should suffer a significant quality drop when applied to a different translation do- main, or to ill-formed sentences. Clearly, met- ric scores computed on partial or wrong syntac- tic/semantic structures will be less informed. But, should this necessarily lead to less reliable eval- uations? In this work, we have analyzed this is- sue by conducting a contrastive empirical study on the …

Content truncated to 500 words