BEER: BEtter Evaluation as Ranking

Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 414–419, Baltimore, Maryland USA, June 26–27, 2014. c⃝2014 Association for Computational Linguistics BEER: BEtter Evaluation as Ranking Miloˇs Stanojevi´c ILLC University of Amsterdam mstanojevic@uva.nl Khalil Sima’an ILLC University of Amsterdam k.simaan@uva.nl Abstract We present the UvA-ILLC submission of the BEER metric to WMT 14 metrics task. BEER is a sentence level metric that can incorporate a large number of features combined in a linear model. Novel con- tributions are (1) efficient tuning of a large number of features for maximizing corre- lation with human system ranking, and (2) novel features that give smoother sentence level scores. 1 Introduction The quality of sentence level (also called segment level) evaluation metrics in machine translation is often considered inferior to the quality of corpus (or system) level metrics. Yet, a sentence level metrics has important advantages as it: 1. provides an informative score to individual translations 2. is assumed by MT tuning algorithms (Hop- kins and May, 2011). 3. facilitates easier statistical testing using sign test or t-test (Collins et al., 2005) We think that the root cause for most of the diffi- culty in creating a good sentence level metric is the sparseness of the features often used. Consider the n-gram counting metrics (BLEU (Papineni et al., 2002)): counts of higher order n-grams are usu- ally rather small, if not zero, when counted at the individual sentence level. Metrics based on such counts are brittle at the sentence level even when they might be good at the corpus level. Ideally we should have features of varying granularity that we can optimize on the actual evaluation task: relative ranking of system outputs. Therefore, in this paper we explore two kinds of less sparse features: Character n-grams are features at the sub-word level that provide evidence for translation ad- equacy - for example whether the stem is cor- rectly translated, Abstract ordering patterns found in tree factor- izations of permutations into Permutation Trees (PETs) (Zhang and Gildea, 2007), in- cluding non-lexical alignment patterns. The BEER metric combines features of both kinds (presented in Section 2). With the growing number of adequacy and or- dering features we need a model that facilitates ef- ficient training. We would like to train for opti- mal Kendall τ correlation with rankings by human evaluators. The models in the literature tackle this problem by 1. training for another similar objective – e.g., tuning for absolute adequacy and fluency scores instead on rankings, or 2. training for rankings directly but with meta- heuristic approaches like hill-climbing, or 3. training for pairwise rankings using learning- to-rank techniques Approach (1) has two disadvantages. One is the inconsistency between the training and the testing objectives. The other, is that absolute rankings are not reliable enough because humans are better at giving relative than absolute judgments (see WMT manual evaluations (Callison-Burch et al., 2007)). Approach (2) does not allow integrating a large number of features which makes it less attractive. Approach (3) allows integration …

Content truncated to 500 words