Automatic Evaluation of Machine Translation Quality Using Longest Common Subsequence and Skip-Bigram Statistics

Automatic Evaluation of Machine Translation Quality Using Longest Com- mon Subsequence and Skip-Bigram Statistics Chin-Yew Lin and Franz Josef Och Information Sciences Institute University of Southern California 4676 Admiralty Way Marina del Rey, CA 90292, USA {cyl,och}@isi.edu Abstract In this paper we describe two new objective automatic evaluation methods for machine translation. The first method is based on long- est common subsequence between a candidate translation and a set of reference translations. Longest common subsequence takes into ac- count sentence level structure similarity natu- rally and identifies longest co-occurring in- sequence n-grams automatically. The second method relaxes strict n-gram matching to skip- bigram matching. Skip-bigram is any pair of words in their sentence order. Skip-bigram co- occurrence statistics measure the overlap of skip-bigrams between a candidate translation and a set of reference translations. The empiri- cal results show that both methods correlate with human judgments very well in both ade- quacy and fluency. 1 Introduction Using objective functions to automatically evalu- ate machine translation quality is not new. Su et al. (1992) proposed a method based on measuring edit distance (Levenshtein 1966) between candidate and reference translations. Akiba et al. (2001) ex- tended the idea to accommodate multiple refer- ences. Nießen et al. (2000) calculated the length- normalized edit distance, called word error rate (WER), between a candidate and multiple refer- ence translations. Leusch et al. (2003) proposed a related measure called position-independent word error rate (PER) that did not consider word posi- tion, i.e. using bag-of-words instead. Instead of error measures, we can also use accuracy measures that compute similarity between candidate and ref- erence translations in proportion to the number of common words between them as suggested by Melamed (1995). An n-gram co-occurrence meas- ure, BLEU, proposed by Papineni et al. (2001) that calculates co-occurrence statistics based on n-gram overlaps have shown great potential. A variant of BLEU developed by NIST (2002) has been used in two recent large-scale machine translation evalua- tions. Recently, Turian et al. (2003) indicated that standard accuracy measures such as recall, preci- sion, and the F-measure can also be used in evalua- tion of machine translation. However, results based on their method, General Text Matcher (GTM), showed that unigram F-measure correlated best with human judgments while assigning more weight to higher n-gram (n > 1) matches achieved similar performance as Bleu. Since unigram matches do not distinguish words in consecutive positions from words in the wrong order, measures based on position-independent unigram matches are not sensitive to word order and sentence level structure. Therefore, systems optimized for these unigram-based measures might generate adequate but not fluent target language. Since BLEU has been used to report the perform- ance of many machine translation systems and it has been shown to correlate well with human judgments, we will explain BLEU in more detail and point out its limitations in the next section. We then introduce a new evaluation method called ROUGE-L that measures sentence-to-sentence similarity based on the longest common subse- quence statistics between a …

Content truncated to 500 words