Evaluate with Confidence Estimation: Machine ranking of translation outputs using grammatical features

Proceedings of the 6th Workshop on Statistical Machine Translation, pages 65–70, Edinburgh, Scotland, UK, July 30–31, 2011. c⃝2011 Association for Computational Linguistics Evaluate with Confidence Estimation: Machine ranking of translation outputs using grammatical features Eleftherios Avramidis, Maja Popovic, David Vilar, Aljoscha Burchardt German Research Center for Artificial Intelligence (DFKI) Language Technology (LT), Berlin, Germany name.surname@dfki.de Abstract We present a pilot study on an evaluation method which is able to rank translation out- puts with no reference translation, given only their source sentence. The system employs a statistical classifier trained upon existing hu- man rankings, using several features derived from analysis of both the source and the tar- get sentences. Development experiments on one language pair showed that the method has considerably good correlation with human ranking when using features obtained from a PCFG parser. 1 Introduction Automatic evaluation metrics for Machine Transla- tion (MT) have mainly relied on analyzing both the MT output against (one or more) reference transla- tions. Though, several paradigms in Machine Trans- lation Research pose the need to estimate the quality through many translation outputs, when no reference translation is given (n-best rescoring of SMT sys- tems, system combination etc.). Such metrics have been known as Confidence Estimation metrics and quite a few projects have suggested solutions on this direction. With our submission to the Shared Task, we allow such a metric to be systematically com- pared with the state-of-the-art reference-aware MT metrics. Our approach suggests building a Confidence Es- timation metric using already existing human judg- ments. This has been motivated by the existence of human-annotated data containing comparisons of the outputs of several systems, as a result of the evaluation tasks run by the Workshops on Statistical Machine Translation (WMT) (Callison-Burch et al., 2008; Callison-Burch et al., 2009; Callison-Burch et al., 2010). This amount of data, which has been freely available for further research, gives an op- portunity for applying machine learning techniques to model the human annotators’ choices. Machine Learning methods over previously released evalua- tion data have been already used for tuning com- plex statistical evaluation metrics (e.g. SVM-Rank in Callison-Burch et al. (2010)). Our proposition is similar, but works without reference translations. We develop a solution of applying machine learning in order to build a statistical classifier that performs similar to the human ranking: it is trained to rank several MT outputs, given analysis of possible qual- itative criteria on both the source and the target side of every given sentence. As qualitative criteria, we use statistical features indicating the quality and the grammaticality of the output. 2 Automatic ranking method 2.1 From Confidence Estimation to ranking Confidence estimation has been seen from the Nat- ural Language Processing (NLP) perspective as a problem of binary classification in order to assess the correctness of a NLP system output. Previ- ous work focusing on Machine Translation includes statistical methods for estimating correctness scores or correctness probabilities, following a rich search over the spectrum of possible features (Blatz et al., 2004a; Ueffing and Ney, …

Content truncated to 500 words