TESLA at WMT 2011: Translation Evaluation and Tunable Metric

Proceedings of the 6th Workshop on Statistical Machine Translation, pages 78–84, Edinburgh, Scotland, UK, July 30–31, 2011. c⃝2011 Association for Computational Linguistics TESLA at WMT 2011: Translation Evaluation and Tunable Metric Daniel Dahlmeier1 and Chang Liu2 and Hwee Tou Ng1,2 1NUS Graduate School for Integrative Sciences and Engineering 2Department of Computer Science, National University of Singapore {danielhe,liuchan1,nght}@comp.nus.edu.sg Abstract This paper describes the submission from the National University of Singapore to the WMT 2011 Shared Evaluation Task and the Tunable Metric Task. Our entry is TESLA in three dif- ferent configurations: TESLA-M, TESLA-F, and the new TESLA-B. 1 Introduction TESLA (Translation Evaluation of Sentences with Linear-programming-based Analysis) was first pro- posed in Liu et al. (2010). The simplest variant, TESLA-M (M stands for minimal), is based on N- gram matching, and utilizes light-weight linguis- tic analysis including lemmatization, part-of-speech tagging, and WordNet synonym relations. TESLA- B (B stands for basic) additionally takes advan- tage of bilingual phrase tables to model phrase syn- onyms. It is a new configuration proposed in this pa- per. The most sophisticated configuration TESLA-F (F stands for full) additionally uses language mod- els and a ranking support vector machine instead of simple averaging. TESLA-F was called TESLA in Liu et al. (2010). In this paper, we rationalize the naming convention by using TESLA to refer to the whole family of metrics. The rest of this paper is organized as follows. Sec- tions 2 to 4 describe the TESLA variants TESLA-M, TESLA-B, and TESLA-F, respectively. Section 5 describes MT tuning with TESLA. Section 6 shows experimental results for the evaluation and the tun- able metric task. The last section concludes the pa- per. 2 TESLA-M The version of TESLA-M used in WMT 2011 is ex- actly the same as in Liu et al. (2010). The descrip- tion is reproduced here for completeness. We consider the task of evaluating machine trans- lation systems in the direction of translating a source language to a target language. We are given a refer- ence translation produced by a professional human translator and a machine-produced system transla- tion. At the highest level, TESLA-M is the arith- metic average of F-measures between bags of N- grams (BNGs). A BNG is a multiset of weighted N-grams. Mathematically, a BNG B consists of tu- ples (bi, bW i ), where each bi is an N-gram and bW i a positive real number representing the weight of bi.is In the simplest case, a BNG contains every N-gram in a translated sentence, and the weights are just the counts of the respective N-grams. However, to em- phasize the content words over the function words, we discount the weight of an N-gram by a factor of 0.1 for every function word in the N-gram. We de- cide whether a word is a function word based on its POS tag. In TESLA-M, the BNGs are extracted in the target language, so we call them bags of target language N-grams (BTNGs). 2.1 Similarity functions To match two BNGs, we first need …

Content truncated to 500 words