A Smorgasbord of Features for Automatic MT Evaluation

Proceedings of the Third Workshop on Statistical Machine Translation, pages 195–198, Columbus, Ohio, USA, June 2008. c⃝2008 Association for Computational Linguistics A Smorgasbord of Features for Automatic MT Evaluation Jes´us Gim´enez and Llu´ıs M`arquez TALP Research Center, LSI Department Universitat Polit`ecnica de Catalunya Jordi Girona Salgado 1–3, E-08034, Barcelona {jgimenez,lluism}@lsi.upc.edu Abstract This document describes the approach by the NLP Group at the Technical University of Cat- alonia (UPC-LSI), for the shared task on Au- tomatic Evaluation of Machine Translation at the ACL 2008 Third SMT Workshop. 1 Introduction Our proposal is based on a rich set of individual metrics operating at different linguistic levels: lex- ical (i.e., on word forms), shallow-syntactic (e.g., on word lemmas, part-of-speech tags, and base phrase chunks), syntactic (e.g., on dependency and con- stituency trees), shallow-semantic (e.g., on named entities and semantic roles), and semantic (e.g., on discourse representations). Although from differ- ent viewpoints, and based on different similarity as- sumptions, in all cases, translation quality is mea- sured by comparing automatic translations against human references. Extensive details on the met- ric set may be found in the IQMT technical manual (Gim´enez, 2007). Apart from individual metrics, we have also applied a simple integration scheme based on uniformly-averaged linear metric combinations (Gim´enez and M`arquez, 2008a). 2 What is new? The main novelty, with respect to the set of metrics presented last year (Gim´enez and M`arquez, 2007), is the incorporation of a novel family of metrics at the properly semantic level. DR metrics ana- lyze similarities between automatic and reference translations by comparing their respective discourse representation structures (DRS), as provided by the the C&C Tools (Clark and Curran, 2004). DRS are essentially a variation of first-order predicate calcu- lus which can be seen as semantic trees. We use three different kinds of metrics: DR-STM Semantic Tree Matching, a la Liu and Gildea (2005), but over DRS instead of over constituency trees. DR-Or-⋆ Lexical overlapping over DRS. DR-Orp-⋆ Morphosyntactic overlapping on DRS. Further details on DR metrics can be found in (Gim´enez and M`arquez, 2008b). 2.1 Improved Sentence Level Behavior Metrics based on deep linguistic analysis rely on automatic processors trained on out-domain data, which may be, thus, prone to error. Indeed, we found out that in many cases, metrics are unable to pro- duce a result due to the lack of linguistic analysis. For instance, in our experiments, for SR metrics, we found that the semantic role labeler was unable to parse 14% of the sentences. In order to improve the recall of these metrics, we have designed two simple variants. Given a linguistic metric x, we define: • xb → by backing off to lexical overlapping, Ol, only when the linguistic processor is not able to produce a linguistic analysis. Other- wise, x score is returned. Lexical scores are conveniently scaled so that they are in a similar range to scores of x. Specifically, we multiply 195 them by the average x score attained over all other test cases for which the parser succeeded. • …

Content truncated to 500 words