BEER 1.1: ILLC UvA submission to metrics and tuning task

Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 396–401, Lisboa, Portugal, 17-18 September 2015. c⃝2015 Association for Computational Linguistics. BEER 1.1: ILLC UvA submission to metrics and tuning task Miloˇs Stanojevi´c ILLC University of Amsterdam m.stanojevic@uva.nl Khalil Sima’an ILLC University of Amsterdam k.simaan@uva.nl Abstract We describe the submissions of ILLC UvA to the metrics and tuning tasks on WMT15. Both submissions are based on the BEER evaluation metric origi- nally presented on WMT14 (Stanojevi´c and Sima’an, 2014a). The main changes introduced this year are: (i) extending the learning-to-rank trained sentence level metric to the corpus level (but still decom- posable to sentence level), (ii) incorporat- ing syntactic ingredients based on depen- dency trees, and (iii) a technique for find- ing parameters of BEER that avoid “gam- ing of the metric” during tuning. 1 Introduction In the 2014 WMT metrics task, BEER turned up as the best sentence level evaluation metric on aver- age over 10 language pairs (Machacek and Bojar, 2014). We believe that this was due to: 1. learning-to-rank - type of training that allows a large number of features and also training on the same objective on which the model is going to be evaluated : ranking of translations 2. dense features - character n-grams and skip- bigrams that are less sparse on the sentence level than word n-grams 3. permutation trees - hierarchical decompo- sition of word order based on (Zhang and Gildea, 2007) A deeper analysis of (2) is presented in (Stano- jevi´c and Sima’an, 2014c) and of (3) in (Stanojevi´c and Sima’an, 2014b). Here we modify BEER by 1. incorporating a better scoring function that give scores that are better scaled 2. including syntactic features and 3. removing the recall bias from BEER . In Section 2 we give a short introduction to BEER after which we move to the innovations for this year in Sections 3, 4 and 5. We show the re- sults from the metric and tuning tasks in Section 6, and conclude in Section 7. 2 BEER basics The model underying the BEER metric is flexible for the integration of an arbitrary number of new features and has a training method that is targeted for producing good rankings among systems. Two other characteristic properties of BEER are its hi- erarchical reordering component and character n- grams lexical matching component. 2.1 Old BEER scoring BEER is essentially a linear model with which the score can be computed in the following way: score(h, r) = � i wi × φi(h, r) = ⃗w · ⃗φ where ⃗w is a weight vector and ⃗φ is a feature vector. 2.2 Learning-to-rank Since the task on which our model is going to be evaluated is ranking translations it comes natu- ral to train the model using learning-to-rank tech- niques. Our training data consists of pairs of “good” and “bad” translations. By using a feature vector ⃗φgood for a good translation and a feature vector ⃗φbad for a bad translation then using the following equations …

Content truncated to 500 words