Regression and Ranking based Optimisation for Sentence Level MT Evaluation

Proceedings of the 6th Workshop on Statistical Machine Translation, pages 123–129, Edinburgh, Scotland, UK, July 30–31, 2011. c⃝2011 Association for Computational Linguistics Regression and Ranking based Optimisation for Sentence Level Machine Translation Evaluation Xingyi Song and Trevor Cohn The Department of Computer Science University of Sheffield Sheffield, S1 4DP. UK {xsong2,t.cohn}@shef.ac.uk Abstract Automatic evaluation metrics are fundamen- tally important for Machine Translation, al- lowing comparison of systems performance and efficient training. Current evaluation met- rics fall into two classes: heuristic approaches, like BLEU, and those using supervised learn- ing trained on human judgement data. While many trained metrics provide a better match against human judgements, this comes at the cost of including lots of features, leading to unwieldy, non-portable and slow metrics. In this paper, we introduce a new trained met- ric, ROSE, which only uses simple features that are easy portable and quick to compute. In addition, ROSE is sentence-based, as op- posed to document-based, allowing it to be used in a wider range of settings. Results show that ROSE performs well on many tasks, such as ranking system and syntactic constituents, with results competitive to BLEU. Moreover, this still holds when ROSE is trained on hu- man judgements of translations into a different language compared with that use in testing. 1 Introduction Human judgements of translation quality are very expensive. For this reason automatic MT evalu- ation metrics are used to as an approximation by comparing predicted translations to human authored references. An early MT evaluation metric, BLEU (Papineni et al., 2002), is still the most commonly used metric in automatic machine translation evalu- ation. However, several drawbacks have been stated by many researchers (Chiang et al., 2008a; Callison- Burch et al., 2006; Banerjee and Lavie, 2005), most notably that it omits recall (substituting this with a penalty for overly short output) and not being easily applied at the sentence level. Later heuristic metrics such as METEOR (Banerjee and Lavie, 2005) and TER (Snover et al., 2006) account for both precision and recall, but their relative weights are difficult to determine manually. In contrast to heuristic metrics, trained met- rics use supervised learning to model directly hu- man judgements. This allows the combination of different features and can better fit specific tasks, such as evaluation focusing more on flu- ency/adequacy/relative ranks or post editing effort. Previous work includes approaches using classifica- tion (Corston-Oliver et al., 2001), regression (Alber- cht and Hwa, 2008; Specia and Gimenez, 2010), and ranking (Duh, 2008). Most of which achieved good results and better correlations with human judg- ments than heuristic baseline methods. Overall automatic metrics must find a balance be- tween several key issues: a) applicability to differ- ent sized texts (documents vs sentences), b) easy of portability to different languages, c) runtime re- quirements and d) correlation with human judge- ment data. Previous work has typically ignored at least one of these issues, e.g., BLEU which applies only to documents (A), trained metrics (Albercht and Hwa, 2008; Specia and Gimenez, 2010) which …

Content truncated to 500 words