Extending Machine Translation Evaluation Metrics with Lexical Cohesion to Document Level

Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 1060–1068, Jeju Island, Korea, 12–14 July 2012. c⃝2012 Association for Computational Linguistics Extending Machine Translation Evaluation Metrics with Lexical Cohesion To Document Level Billy T. M. Wong and Chunyu Kit Department of Chinese, Translation and Linguistics City University of Hong Kong 83 Tat Chee Avenue, Kowloon, Hong Kong SAR, P. R. China {tmwong,ctckit}@cityu.edu.hk Abstract This paper proposes the utilization of lexical cohesion to facilitate evaluation of machine translation at the document level. As a linguis- tic means to achieve text coherence, lexical cohesion ties sentences together into a mean- ingfully interwoven structure through words with the same or related meaning. A compar- ison between machine and human translation is conducted to illustrate one of their critical distinctions that human translators tend to use more cohesion devices than machine. Various ways to apply this feature to evaluate machine- translated documents are presented, including one without reliance on reference translation. Experimental results show that incorporating this feature into sentence-level evaluation met- rics can enhance their correlation with human judgements. 1 Introduction Machine translation (MT) has benefited a lot from the advancement of automatic evaluation in the past decade. To a certain degree, its progress is also con- fined to the limitations of evaluation metrics in use. Most efforts devoted to evaluate the quality of MT output so far have still focused on the sentence level without sufficient attention to how a larger text is structured. This is notably reflected in the represen- tative MT evaluation metrics, such as BLEU (Pap- ineni et al., 2002), METEOR (Banerjee and Lavie, 2005) and TER (Snover et al., 2006), that adopt a sentence-by-sentence fashion to score MT outputs. The evaluation result for a document by any of them is usually a simple average of its sentence scores. A drawback of this kind of sentence-based evaluation is the neglect of document structure. There is no guarantee for the coherence of a text if it is produced by simply putting together stand-alone sentences, no matter how well-translated, without adequate inter- sentential connection. As a consequence, MT sys- tem optimized this way to any of these metrics can only have a very dim chance of producing translated document that reads as natural as human writing. The accuracy of MT output at the document level is particularly important to MT users, for they care about the overall meaning of a text in question more than the grammatical correctness of each sentence (Visser and Fuji, 1996). Post-editors particularly need to ensure the quality of a whole document of MT output when revising its sentences. The con- nectivity of sentences is surely a significant factor contributing to the understandability of a text as a whole. This paper studies the inter-sentential linguistic features of cohesion and coherence and presents plausible ways to incorporate them into the sentence-based metrics to support MT evaluation at the document level. In the Framework for MT Eval- uation in …

Content truncated to 500 words