Normalized Compression Distance Based Measures for MetricsMATR 2010

Proceedings of the Joint 5th Workshop on Statistical Machine Translation and MetricsMATR, pages 343–348, Uppsala, Sweden, 15-16 July 2010. c⃝2010 Association for Computational Linguistics Normalized Compression Distance Based Measures for MetricsMATR 2010 Marcus Dobrinkat and Jaakko V¨ayrynen and Tero Tapiovaara Adaptive Informatics Research Centre Aalto University School of Science and Technology P.O. Box 15400, FI-00076 Aalto, Finland {marcus.dobrinkat,jaakko.j.vayrynen,tero.tapiovaara}@tkk.fi Kimmo Kettunen Kymenlaakso University of Applied Sciences P.O. Box 9, FI-48401 Kotka, Finland Kimmo.kettunen@kyamk.fi Abstract We present the MT-NCD and MT-mNCD machine translation evaluation metrics as submission to the machine transla- tion evaluation shared task (MetricsMATR 2010). The metrics are based on nor- malized compression distance (NCD), a general information theoretic measure of string similarity, and evaluated against hu- man judgments from the WMT08 shared task. The experiments show that 1) our metric improves correlation to hu- man judgments by using flexible match- ing, 2) segment replication is effective, and 3) our NCD-inspired method for mul- tiple references indicates improved results. Generally, the proposed MT-NCD and MT-mNCD methods correlate competi- tively with human judgments compared to commonly used machine translations eval- uation metrics, for instance, BLEU. 1 Introduction The quality of automatic machine translation (MT) evaluation metrics plays an important role in the development of MT systems. Human eval- uation would no longer be necessary if automatic MT metrics correlated perfectly with manual judg- ments. Besides high correlation with human judg- ments of translation quality, a good metric should be language independent, fast to compute and sen- sitive enough to reliably detect small improve- ments in MT systems. Recently there have been some experiments with normalized compression distance (NCD) as a method for automatic evaluation of machine trans- lation. NCD is a general string similarity measure that has been useful for clustering in various tasks (Cilibrasi and Vitanyi, 2005). Parker (2008) introduced BADGER, a machine translation evaluation metric that uses NCD to- gether with a language independent word normal- ization method. Kettunen (2009) independently applied NCD to the direct evaluation of transla- tions. He showed with a small corpus of three lan- guage pairs that the scores of NCD and METEOR (v0.6) from translations of 10–12 MT systems were highly correlated. V¨ayrynen et al. (2010) have extended the work by showing that NCD can be used to rank transla- tions of different MT systems so that the ranking order correlates with human rankings at the same level as BLEU (Papineni et al., 2001). For trans- lations into English, NCD had an overall system- level correlation of 0.66 whereas the best method, ULC had an overall correlation of 0.76, and BLEU had an overall correlation of 0.65. NCD presents a viable alternative to the de facto standard BLEU. Both metrics are language independent, simple and efficient to compute. However, NCD is a general measure of similarity that has been ap- plied in many domains. More advanced meth- ods achieve better correlation with human judg- ments, but typically use additional language spe- cific linguistic resources. Dobrinkat et al. (2010) experimented with relaxed word matching, …

Content truncated to 500 words