Sentence-level MT evaluation without reference translations: beyond language modeling

EAMT 2005 Conference Proceedings 103 Sentence-level MT evaluation without reference translations: Beyond language modeling Michael Gamon, Anthony Aue, Martine Smets Microsoft Research, Microsoft Customer Service and Support mgamon|anthaue|msmets@microsoft.com Abstract. In this paper we investigate the possibility of evaluating MT quality and fluency at the sentence level in the absence of reference translations. We measure the correlation between automatically-generated scores and human judgments, and we evaluate the per- formance of our system when used as a classifier for identifying highly dysfluent and ill- formed sentences. We show that we can substantially improve on the correlation between language model perplexity scores and human judgment by combining these perplexity scores with class probabilities from a machine-learned classifier. The classifier uses linguis- tic features and has been trained to distinguish human translations from machine transla- tions. We show that this approach also performs well in identifying dysfluent sentences. 1. Introduction It is widely recognized that the automatic eval- uation of machine translation quality is crucial for at least two major tasks: • comparisons of different systems on the same or similar data sets • system-internal evaluations in the practical deployment of MT systems There is a rapidly-growing body of literature on automatic metrics for the first scenario, starting with the BLEU metric (Papineni et al. 2002) and the NIST metric (Doddington 2002). Crucially, the BLEU metric and its relatives (e.g. Lin and Och 2004b, Babych and Hartley 2004, Soricut and Brill 2004) rely on one or more human ref- erence translations for each machine-translated sentence to be evaluated. These metrics are all based on the idea that the more shared sub- strings the machine-translated sentence has with the human reference translation(s), the better the translation is. It has been shown that BLEU scores, despite their shortcomings, correlate sur- prisingly well with human judgment (Coughlin 2003). The BLEU metric and its relatives are typically computed for fixed multi-sentence test sets in order to track the performance of MT systems over time and to compare different MT systems with respect to these test sets. Moving from multi-sentence evaluation to single-sentence and even word-level evaluation, Blatz et al. (2004) survey a number of ap- proaches to the estimation of confidence. The training data for their experiments consist of machine translations and human reference trans- lations. This work uses naive Bayes and multi- layer perceptrons for classification. For the system-internal evaluation of de- ployed MT systems, Quirk (2004) uses a small (350 sentence) corpus of machine translations that have been annotated for translation quality by human annotators. He represents translated sentences as feature vectors, training a classifier to emulate the human scoring. The features he uses include sentence perplexity according to a trigram language model (LM) of a training cor- pus, source sentence features such as length, translation features such as number and size of mappings, whether a translation comes from a learned mapping or from a dictionary, and ti- lings of source and translated sentences with re- spect to the training corpus. Quirk demonstrates that this approach …

Content truncated to 500 words