Capturing Lexical Variation in MT Evaluation Using Automatically Built Sense-Cluster Inventories

Capturing Lexical Variation in MT Evaluation Using Automatically Built Sense-Cluster Inventories Marianna Apidianaki, Yifan He, and Andy Way NCLT/CNGL School of Computing, Dublin City University Dublin 9, Ireland {mapidianaki, yhe, away}@computing.dcu.ie Abstract. The strict character of most of the existing Machine Translation (MT) evaluation metrics does not permit them to capture lexical variation in translation. However, a central issue in MT evaluation is the high correlation that the metrics should have with human judg- ments of translation quality. In order to achieve a higher correlation, the identification of sense correspondences between the compared translations becomes really important. Given that most metrics are looking for exact correspondences, the evaluation results are often mis- leading concerning translation quality. Apart from that, existing metrics do not permit one to make a conclusive estimation of the impact of Word Sense Disambiguation techniques into MT systems. In this paper, we show how information acquired by an unsupervised semantic analysis method can be used to render MT evaluation more sensitive to lexical semantics. The sense inventories built by this data-driven method are incorporated into METEOR: they replace WordNet for evaluation in English and render METEOR’s synonymy module operable in French. The evaluation results demonstrate that the use of these inventories gives rise to an increase in the number of matches and the correlation with human judgments of translation quality, compared to precision-based metrics. Keywords: Machine Translation, evaluation, synonymy, sense clustering 1 Introduction The majority of the existing Machine Translation (MT) evaluation metrics look for exact surface correspondences between the compared translations. They are mostly based on the strict precision criterion, which does not account for the semantic similarity of words found in the hypothesis and the reference. Given that evaluation scores often do not reflect the quality of translation, there is a growing tendency towards increasing the correlation of the metrics with human judgments of translation quality. An important factor determining this correlation is the identification of sense correspondences between the hypothesis and the reference, which may exist even if the words used in the translations differ. Capturing this type of correspondence would also allow a more conclusive estimation of the impact of WSD techniques on MT systems than is possible with the current evaluation metrics (Callison-Burch et al., 2006; Carpuat and Wu, 2005; Chan et al., 2007). In this paper, we show how variation at the unigram level can be captured during evaluation using information induced from parallel corpora by an unsupervised sense induction method. This method generates bilingual semantic inventories, where the senses of the words of one language are described by clusters of their semantically similar translation equivalents (TEs) in another language. These sense-clusters, which are similar to WordNet1 synsets, can serve to capture cor- respondences between synonymous words found in the compared translations. Copyright 2009 by Marianna Apidianaki, Yifan He, and Andy Way 1 http://wordnet.princeton.edu/ 53 23rd Pacific Asia Conference on Language, Information and Computation, pages 53–62 The structure of the paper is as follows: in the next section, we present how …

Content truncated to 500 words