Fitting Sentence Level Translation Evaluation with Many Dense Features

Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 202–206, October 25-29, 2014, Doha, Qatar. c⃝2014 Association for Computational Linguistics Fitting Sentence Level Translation Evaluation with Many Dense Features Miloˇs Stanojevi´c and Khalil Sima’an Institute for Logic, Language and Computation University of Amsterdam Science Park 107, 1098 XG Amsterdam, The Netherlands {m.stanojevic,k.simaan}@uva.nl Abstract Sentence level evaluation in MT has turned out far more difficult than corpus level evaluation. Existing sentence level metrics employ a lim- ited set of features, most of which are rather sparse at the sentence level, and their intricate models are rarely trained for ranking. This pa- per presents a simple linear model exploiting 33 relatively dense features, some of which are novel while others are known but seldom used, and train it under the learning-to-rank frame- work. We evaluate our metric on the stan- dard WMT12 data showing that it outperforms the strong baseline METEOR. We also ana- lyze the contribution of individual features and the choice of training data, language-pair vs. target-language data, providing new insights into this task. 1 Introduction Evaluating machine translation (MT) output at the sen- tence/ segment level has turned out far more challeng- ing than corpus/ system level. Yet, sentence level evaluation can be useful because it allows fast, fine- grained analysis of system performance on individual sentences. It is instructive to contrast two widely used metrics, METEOR (Michael Denkowski and Alon Lavie, 2014) and BLEU (Papineni et al., 2002), on sentence level evaluation. METEOR constantly shows better corre- lation with human ranking than BLEU (Papineni et al., 2002). Arguably, this shows that sentence level evaluation demands finer grained and trainable models over less sparse features. Ngrams, the core of BLEU, are sparse at the sentence level, and a mismatch for longer ngrams implies that BLEU falls back on shorter ngrams. In contrast, METEOR has a trainable model and incorporates a small, yet wider set of features that are less sparse than ngrams. We think that METEOR’s features and its training approach only suggest that sen- tence level evaluation should be treated as a modelling challenge. This calls for questions such as what model, what features and what training objective are better suited for modelling sentence level evaluation. We start out by explicitly formulating sentence level evaluation as the problem of ranking a set of compet- ing hypothesis. Given data consisting of human ranked system outputs, the problem then is to formulate an easy to train model for ranking. One particular exist- ing approach (Ye et al., 2007) looks especially attrac- tive because we think it meshes well with a range of effective techniques for learning-to-rank (Li, 2011). We deliberately select a linear modelling approach inspired by RankSVM (Herbrich et al., 1999), which is easily trainable for ranking and allows analysis of the individual contributions of features. Besides presenting a new metric and a set of known, but also a set of novel features, we target three questions of interest to the MT community: • …

Content truncated to 500 words