A Machine Learning Approach to the Automatic Evaluation of Machine Translation

A machine learning approach to the automatic evaluation of machine translation Simon Corston-Oliver, Michael Gamon and Chris Brockett Microsoft Research One Microsoft Way Redmond WA 98052, USA {simonco, mgamon, chrisbkt}@microsoft.com Abstract We present a machine learning approach to evaluating the well- formedness of output of a machine translation system, using classifiers that learn to distinguish human reference translations from machine translations. This approach can be used to evaluate an MT system, tracking improvements over time; to aid in the kind of failure analysis that can help guide system development; and to select among alternative output strings. The method presented is fully automated and independent of source language, target language and domain. 1 Introduction Human evaluation of machine translation (MT) output is an expensive process, often prohibitively so when evaluations must be performed quickly and frequently in order to measure progress. This paper describes an approach to automated evaluation designed to facilitate the identification of areas for investigation and improvement. It focuses on evaluating the wellformedness of output and does not address issues of evaluating content transfer. Researchers are now applying automated evaluation in MT and natural language generation tasks, both as system-internal goodness metrics and for the assessment of output. Langkilde and Knight (1998), for example, employ n-gram metrics to select among candidate outputs in natural language generation, while Ringger et al. (2001) use n- gram perplexity to compare the output of MT systems. Su et al. (1992), Alshawi et al. (1998) and Bangalore et al. (2000) employ string edit distance between reference and output sentences to gauge output quality for MT and generation. To be useful to researchers, however, assessment must provide linguistic information that can guide in identifying areas where work is required. (See Nyberg et al., 1994 for useful discussion of this issue.) The better the MT system, the more its output will resemble human-generated text. Indeed, MT might be considered a solved problem should it ever become impossible to distinguish automated output from human translation. We have observed that in general humans can easily and reliably categorize a sentence as either machine- or human-generated. Moreover, they can usually justify their decision. This observation suggests that evaluation of the wellformedness of output sentences can be treated as a classification problem: given a sentence, how accurately can we predict whether it has been translated by machine? In this paper we cast the problem of MT evaluation as a machine learning classification task that targets both linguistic features and more abstract features such as n- gram perplexity. 2 Data Our corpus consists of 350,000 aligned Spanish- English sentence pairs taken from published computer software manuals and online help documents. We extracted 200,000 English sentences for building language models to evaluate per-sentence perplexity. From the remainder of the corpus, we extracted 100,000 aligned sentence pairs. The Spanish sentences in this latter sample were then translated by the Microsoft machine translation system, which was trained on documents from this domain (Richardson et al., 2001). This yielded a set of 200,000 English …

Content truncated to 500 words