Extending the BLEU MT Evaluation Method with Frequency Weightings

Extending the BLEU MT Evaluation Method with Frequency Weightings Bogdan Babych Centre for Translation Studies University of Leeds Leeds, LS2 9JT, UK bogdan@comp.leeds.ac.uk Anthony Hartley Centre for Translation Studies University of Leeds Leeds, LS2 9JT, UK a.hartley@leeds.ac.uk Abstract We present the results of an experiment on extending the automatic method of Machine Translation evaluation BLUE with statistical weights for lexical items, such as tf.idf scores. We show that this extension gives additional information about evaluated texts; in particular it al- lows us to measure translation Adequacy, which, for statistical MT systems, is often overestimated by the baseline BLEU method. The proposed model uses a sin- gle human reference translation, which increases the usability of the proposed method for practical purposes. The model suggests a linguistic interpretation which relates frequency weights and human in- tuition about translation Adequacy and Fluency. 1. Introduction Automatic methods for evaluating different as- pects of MT quality – such as Adequacy, Fluency and Informativeness – provide an alternative to an expensive and time-consuming process of human MT evaluation. They are intended to yield scores that correlate with human judgments of translation quality and enable systems (machine or human) to be ranked on this basis. Several such automatic methods have been proposed in recent years. Some of them use human reference translations, e.g., the BLEU method (Papineni et al., 2002), which is based on comparison of N-gram models in MT output and in a set of hu- man reference translations. However, a serious problem for the BLEU method is the lack of a model for relative impor- tance of matched and mismatched items. Words in text usually carry an unequal informational load, and as a result are of differing importance for translation. It is reasonable to expect that the choices of right translation equivalents for certain key items, such as expressions denoting principal events, event participants and relations in a text are more important in the eyes of human evalua- tors then choices of function words and a syntac- tic perspective for sentences. Accurate rendering of these key items by an MT system boosts the quality of translation. Therefore, at least for evaluation of translation Adequacy (Fidelity), the proper choice of translation equivalents for im- portant pieces of information should count more than the choice of words which are used for structural purposes and without a clear translation equivalent in the source text. (The latter may be more important for Fluency evaluation). The problem of different significance of N- gram matches is related to the issue of legitimate variation in human translations, when certain words are less stable than others across inde- pendently produced human translations. BLEU accounts for legitimate translation variation by using a set of several human reference transla- tions, which are believed to be representative of several equally acceptable ways of translating any source segment. This is motivated by the need not to penalise deviations from the set of N- grams in a single reference, although the re- quirement of multiple human references makes …

Content truncated to 500 words