Human Evaluation of Machine Translation Through Binary System Comparisons

Proceedings of the Second Workshop on Statistical Machine Translation, pages 96–103, Prague, June 2007. c⃝2007 Association for Computational Linguistics Human Evaluation of Machine Translation Through Binary System Comparisons David Vilar, Gregor Leusch and Hermann Ney Lehrstuhl f¨ur Informatik 6 RWTH Aachen University D-52056 Aachen, Germany {vilar,leusch,ney}@cs.rwth-aachen.de Rafael E. Banchs D. of Signal Theory and Communications Universitat Polit`ecnica de Catalunya 08034 Barcelona, Spain rbanchs@gps.tsc.upc.edu Abstract We introduce a novel evaluation scheme for the human evaluation of different machine translation systems. Our method is based on direct comparison of two sentences at a time by human judges. These binary judg- ments are then used to decide between all possible rankings of the systems. The ad- vantages of this new method are the lower dependency on extensive evaluation guide- lines, and a tighter focus on a typical eval- uation task, namely the ranking of systems. Furthermore we argue that machine transla- tion evaluations should be regarded as sta- tistical processes, both for human and au- tomatic evaluation. We show how confi- dence ranges for state-of-the-art evaluation measures such as WER and TER can be computed accurately and efficiently without having to resort to Monte Carlo estimates. We give an example of our new evaluation scheme, as well as a comparison with classi- cal automatic and human evaluation on data from a recent international evaluation cam- paign. 1 Introduction Evaluation of machine translation (MT) output is a difficult and still open problem. As in other natu- ral language processing tasks, automatic measures which try to asses the quality of the translation can be computed. The most widely known are the Word Error Rate (WER), the Position independent word Error Rate (PER), the NIST score (Dodding- ton, 2002) and, especially in recent years, the BLEU score (Papineni et al., 2002) and the Translation Er- ror Rate (TER) (Snover et al., 2005). All of the- ses measures compare the system output with one or more gold standard references and produce a nu- merical value (score or error rate) which measures the similarity between the machine translation and a human produced one. Once such reference transla- tions are available, the evaluation can be carried out in a quick, efficient and reproducible manner. However, automatic measures also have big dis- advantages; (Callison-Burch et al., 2006) describes some of them. A major problem is that a given sen- tence in one language can have several correct trans- lations in another language and thus, the measure of similarity with one or even a small amount of ref- erence translations will never be flexible enough to truly reflect the wide range of correct possibilities of a translation. 1 This holds in particular for long sen- tences and wide- or open-domain tasks like the ones dealt with in current MT projects and evaluations. If the actual quality of a translation in terms of usefulness for human users is to be evaluated, human evaluation needs to be carried out. This is however a costly and very time-consuming process. In this work we present …

Content truncated to 500 words