MAXSIM: A Maximum Similarity Metric for Machine Translation Evaluation

Proceedings of ACL-08: HLT, pages 55–62, Columbus, Ohio, USA, June 2008. c⃝2008 Association for Computational Linguistics MAXSIM: A Maximum Similarity Metric for Machine Translation Evaluation Yee Seng Chan and Hwee Tou Ng Department of Computer Science National University of Singapore Law Link, Singapore 117590 {chanys, nght}@comp.nus.edu.sg Abstract We propose an automatic machine translation (MT) evaluation metric that calculates a sim- ilarity score (based on precision and recall) of a pair of sentences. Unlike most metrics, we compute a similarity score between items across the two sentences. We then find a maxi- mum weight matching between the items such that each item in one sentence is mapped to at most one item in the other sentence. This general framework allows us to use arbitrary similarity functions between items, and to in- corporate different information in our com- parison, such as n-grams, dependency rela- tions, etc. When evaluated on data from the ACL-07 MT workshop, our proposed metric achieves higher correlation with human judge- ments than all 11 automatic MT evaluation metrics that were evaluated during the work- shop. 1 Introduction In recent years, machine translation (MT) research has made much progress, which includes the in- troduction of automatic metrics for MT evaluation. Since human evaluation of MT output is time con- suming and expensive, having a robust and accurate automatic MT evaluation metric that correlates well with human judgement is invaluable. Among all the automatic MT evaluation metrics, BLEU (Papineni et al., 2002) is the most widely used. Although BLEU has played a crucial role in the progress of MT research, it is becoming evident that BLEU does not correlate with human judgement well enough, and suffers from several other deficien- cies such as the lack of an intuitive interpretation of its scores. During the recent ACL-07 workshop on statis- tical MT (Callison-Burch et al., 2007), a total of 11 automatic MT evaluation metrics were evalu- ated for correlation with human judgement. The re- sults show that, as compared to BLEU, several re- cently proposed metrics such as Semantic-role over- lap (Gimenez and Marquez, 2007), ParaEval-recall (Zhou et al., 2006), and METEOR (Banerjee and Lavie, 2005) achieve higher correlation. In this paper, we propose a new automatic MT evaluation metric, MAXSIM, that compares a pair of system-reference sentences by extracting n-grams and dependency relations. Recognizing that differ- ent concepts can be expressed in a variety of ways, we allow matching across synonyms and also com- pute a score between two matching items (such as between two n-grams or between two dependency relations), which indicates their degree of similarity with each other. Having weighted matches between items means that there could be many possible ways to match, or link items from a system translation sentence to a reference translation sentence. To match each sys- tem item to at most one reference item, we model the items in the sentence pair as nodes in a bipartite graph and use the Kuhn-Munkres algorithm (Kuhn, 1955; Munkres, 1957) to find a maximum weight matching (or …

Content truncated to 500 words