Diagnostic Evaluation of Machine Translation Systems Using Automatically Constructed Linguistic Check-Points

Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008), pages 1121–1128 Manchester, August 2008 Diagnostic Evaluation of Machine Translation Systems Using Auto- matically Constructed Linguistic Check-Points Ming Zhou1, Bo Wang2, Shujie Liu2, Mu Li1, Dongdong Zhang1, Tiejun Zhao2 1Microsoft Research Asia Beijing, China {mingzhou,muli,dozhang} @microsoft.com 2Harbin Institute of Technology Harbin, China {bowang,Shujieliu,tjzhao} @mtlab.hit.edu.cn Abstract We present a diagnostic evaluation plat- form which provides multi-factored eval- uation based on automatically con- structed check-points. A check-point is a linguistically motivated unit (e.g. an am- biguous word, a noun phrase, a verb~obj collocation, a prepositional phrase etc.), which are pre-defined in a linguistic tax- onomy. We present a method that auto- matically extracts check-points from pa- rallel sentences. By means of check- points, our method can monitor a MT system in translating important linguistic phenomena to provide diagnostic evalua- tion. The effectiveness of our approach for diagnostic evaluation is verified through experiments on various types of MT systems. 1 Introduction Automatic MT evaluation is a crucial issue for MT system developers. The state-of-the-art me- thods for automatic MT evaluation are using an n-gram based metric represented by BLEU (Pa- pineni et al., 2002) and its variants. Ever since its invention, the BLEU score has been a widely accepted benchmark for MT system evaluation. Nevertheless, the research community has been aware of the deficiencies of the BLEU metric (Callison-Burch et al., 2006). For instance, BLEU fails to sufficiently capture the vitality of natural languages: all grams of a sentence are © 2008. Licensed under the Creative Commons Attribution- Noncommercial-Share Alike 3.0 Unported license (http://creativecommons.org/licenses/by-nc-sa/3.0/). Some rights reserved. treated equally ignoring their linguistic signific- ance; only consecutive grams are considered ig- noring the skipped grams of certain linguistic relations; candidate translation gets acknowl- edged only if it uses exactly the same lexicon as the reference ignoring the variation in lexical choice. Furthermore, BLEU is useful for opti- mizing and improving statistical MT systems but it has shown to be ineffective in comparing sys- tems with different architectures (e.g., rule-based vs. phrase-based) (Callison-Burch et al., 2006). Another common deficiency of the state-of- the-art evaluation approaches is that they cannot clearly inform MT developers on the detailed strengths and flaws of an MT system, and there- fore there is no way for us to understand the ca- pability of certain modules of an MT system, and the capability of translating certain kinds of lan- guage phenomena. For the purpose of system development, MT developers need a diagnostic evaluation approach to provide the feedback on the translation ability of an MT system with re- gard to various important linguistic phenomena. We propose a novel diagnostic evaluation ap- proach. Instead of assigning a general score to an MT system we evaluate the capability of the sys- tem in handling various important linguistic test cases called Check-Points. A check-point is a linguistically motivated unit, (e.g. an ambiguous word, a noun phrase, a verb~obj collocation, a prepositional phrase etc.) which are pre-defined in a linguistic taxonomy for diagnostic evalua- …

Content truncated to 500 words