TESLA: Translation Evaluation of Sentences with Linear-Programming-Based Analysis

Proceedings of the Joint 5th Workshop on Statistical Machine Translation and MetricsMATR, pages 354–359, Uppsala, Sweden, 15-16 July 2010. c⃝2010 Association for Computational Linguistics TESLA: Translation Evaluation of Sentences with Linear-programming-based Analysis Chang Liu1 and Daniel Dahlmeier2 and Hwee Tou Ng1,2 1Department of Computer Science, National University of Singapore 2NUS Graduate School for Integrative Sciences and Engineering {liuchan1,danielhe,nght}@comp.nus.edu.sg Abstract We present TESLA-M and TESLA, two novel automatic machine translation eval- uation metrics with state-of-the-art perfor- mances. TESLA-M builds on the suc- cess of METEOR and MaxSim, but em- ploys a more expressive linear program- ming framework. TESLA further exploits parallel texts to build a shallow seman- tic representation. We evaluate both on the WMT 2009 shared evaluation task and show that they outperform all participating systems in most tasks. 1 Introduction In recent years, many machine translation (MT) evaluation metrics have been proposed, exploiting varying amounts of linguistic resources. Heavyweight linguistic approaches including RTE (Pado et al., 2009) and ULC (Giménez and Màrquez, 2008) performed the best in the WMT 2009 shared evaluation task. They exploit an ex- tensive array of linguistic features such as parsing, semantic role labeling, textual entailment, and dis- course representation, which may also limit their practical applications. Lightweight linguistic approaches such as ME- TEOR (Banerjee and Lavie, 2005), MaxSim (Chan and Ng, 2008), wpF and wpBleu (Popovi´c and Ney, 2009) exploit a limited range of linguis- tic information that is relatively cheap to acquire and to compute, including lemmatization, part-of- speech (POS) tagging, and synonym dictionaries. Non-linguistic approaches include BLEU (Pap- ineni et al., 2002) and its variants, TER (Snover et al., 2006), among others. They operate purely at the surface word level and no linguistic resources are required. Although still very popular with MT researchers, they have generally shown inferior performances than the linguistic approaches. We believe that the lightweight linguistic ap- proaches are a good compromise given the current state of computational linguistics research and re- sources. In this paper, we devise TESLA-M and TESLA, two lightweight approaches to MT eval- uation. Specifically: (1) the core features are F- measures derived by matching bags of N-grams; (2) both recall and precision are considered, with more emphasis on recall; and (3) WordNet syn- onyms feature prominently. The main novelty of TESLA-M compared to METEOR and MaxSim is that we match the N- grams under a very expressive linear programming framework, which allows us to assign weights to the N-grams. This is in contrast to the greedy ap- proach of METEOR, and the more restrictive max- imum bipartite matching formulation of MaxSim. In addition, we present a heavier version TESLA, which combines the features using a lin- ear model trained on development data, making it easy to exploit features not on the same scale, and leaving open the possibility of domain adapta- tion. It also exploits parallel texts of the target lan- guage with other languages as a shallow semantic representation, which allows us to model phrase synonyms and idioms. In contrast, METEOR and MaxSim …

Content truncated to 500 words