The Parameter-Optimized ATEC Metric for MT Evaluation

Proceedings of the Joint 5th Workshop on Statistical Machine Translation and MetricsMATR, pages 360–364, Uppsala, Sweden, 15-16 July 2010. c⃝2010 Association for Computational Linguistics The Parameter-optimized ATEC Metric for MT Evaluation Billy T-M Wong Chunyu Kit Department of Chinese, Translation and Linguistics City University of Hong Kong Tat Chee Avenue, Kowloon, Hong Kong {ctbwong, ctckit}@cityu.edu.hk Abstract This paper describes the latest version of the ATEC metric for automatic MT evaluation, with parameters optimized for word choice and word order, the two fundamental features of language that the metric relies on. The former is assessed by matching at various linguistic levels and weighting the informa- tiveness of both matched and unmatched words. The latter is quantified in term of word position and information flow. We also discuss those aspects of language not yet covered by other existing evaluation metrics but carefully considered in the formulation of our metric. 1 Introduction It is recognized that the proposal of the BLEU metric (Papineni et al., 2002) has piloted a para- digm evolution to MT evaluation. It provides a computable solution to the task and turns it into an engineering problem of measuring text simi- larity and simulating human judgments of trans- lation quality. Related studies in recent years have extensively revealed more essential charac- teristics of BLEU, including its strengths and weaknesses. This has aroused the proposal of different new evaluation metrics aimed at ad- dressing such weaknesses so as to find some oth- er hopefully better alternatives for the task. Ef- fort in this direction brings up some advanced metrics such as METEOR (Banerjee and Lavie, 2005) and TERp (Snover et al., 2009) that seem to have already achieved considerably strong correlations with human judgments. Nevertheless, few metrics have really nurtured our understand- ing of possible parameters involved in our lan- guage comprehension and text quality judgment. This inadequacy limits, inevitably, the applica- tion of the existing metrics. The ATEC metric (Wong and Kit, 2008) was developed as a response to this inadequacy, with a focus to account for the process of human comprehension of sentences via two fundamental features of text, namely word choice and word order. It integrates various explicit measures for these two features in order to provide an intuitive and informative evaluation result. Its previous version (Wong and Kit, 2009b) has already illu- strated a highly comparable performance to the few state-of-the-art evaluation metrics, showing a great improvement over its initial version for participation in MetricsMATR081. It is also ap- plied to evaluate online MT systems for legal translation, to examine its applicability for lay users’ use to select appropriate MT systems (Wong and Kit, 2009a). In this paper we describe the formulation of ATEC, including its new features and optimiza- tion of parameters. In particular we will discuss how the design of this metric can complement the inadequacies of other metrics in terms of its treatment of word choice and word order and its utilization of multiple references in the evalua- tion process. 2 The ATEC Metric …

Content truncated to 500 words