A Visualization method for machine translation evaluation results

A Visualization method for machine translation evaluation results Jian-Min YAO1,2, Yun-Qian qu1, Qiao-Ming Zhu1,Jing Zhang3 1School of Computer Science and Technology, Soochow University, Suzhou 215006, China 2School of Computer Science and Engineering, Southeast University, Nanjing 210096, China 3 School of Computer Science, South China University of Technology, Guangzhou 510641, China E-mail: { jyao, quyq, qmzhu}@suda.edu.cn, zhjing@scut.edu.cn Abstract. To make it easier to understand the machine translation evaluation results, a curve is utilized to stand for the performance of a machine translation system. The position of the curve in the graph depicts the quality of the system. The upper left curve stands for higher translation quality. System clustering is made and its dendrogram illustrates the quality difference between systems. These two methods visualize the machine translation evaluation results. Keywords: machine translation, visualization, system clustering 1 Introduction Machine translation evaluation activities have accompanied the MT research and system development. The ALPAC report is the first historical MT evaluation activity. In 1990s, because of the prosperity of machine translation research, great amount of evaluation activities are carried out according to the intelligibility and fidelity metrics followed DARPA methodology. The ISLE takes a software engineering point of view, which focuses on how an MT system serves the follow-on human processing rather than on what it is unlikely to do well. Since manual evaluation is labor-intensive and time-consuming, many researchers are making efforts towards reliable automatic MT evaluation methods. A problem is that the methods can not be characteristic by its precision and recall as in other natural language processing activities such as POS tagging or phrase identification[4]. A new quality system is necessary. This paper aims for the better illustration of machine translation evaluation results and ensemble of different evaluation methods. 2 Related work Evaluation has not been a very powerful aid in machine translation research until it is automated. Now different heuristics are employed for automatic MT evaluation. This section gives a brief review of main automatic MT evaluation methods and study on performance of these methods. Some automatic methods focus on specific syntactic features for translation evaluation. Jones (2000) utilizes linguistic information such as balance of parse trees, N-grams, semantic co-occurrence and so on as indicators of translation quality[1]. Brew C (1994) compares human rankings and automatic measures to decide the translation quality, whose criteria involve word frequency, POS tagging distribution and other text features[2]. Another type of evaluation method involves comparison of the translation result with human translations. Yasuda (2001) evaluates the translation output by measuring the similarity between the translation output and translation answer candidates from a parallel corpus[3]. Akiba (2001) uses multiple edit distances to automatically rank machine translation output by translation examples[4]. While the IBM BLEU method[5] and the NIST method[6] compare MT output with expert reference translations in terms of the statistics of word N-grams. The GTM method [7] adopts the maximum matching size of the translation and reference as similarity measure for score. Another path of MTE is based on test suites. A weighted average of …

Content truncated to 500 words