An Automatic Evaluation Method for Localization Oriented Lexicalised EBMT System

An Automatic Evaluation Method for Localization Oriented Lexicalised EBMT System Jianmin Yao+, Ming Zhou++, Tiejun Zhao+, Hao Yu+, Sheng Li+ +School of Computer Science and Technology Harbin Institute of Technology, Harbin, China, 150001 {james, tjzhao, yu, shengli}@mtlab.hit.edu.cn ++Natural Language Computing Group Microsoft Research Asia Beijing, China, 100080 Mingzhou@microsoft.com Abstract To help developing a localization oriented EBMT system, an automatic machine translation evaluation method is implemented which adopts edit distance, cosine correlation and Dice coefficient as criteria. Experiment shows that the evaluation method distinguishes well between “good” translations and “bad” ones. To prove that the method is consistent with human evaluation, 6 MT systems are scored and compared. Theoretical analysis is made to validate the experimental results. Correlation coefficient and significance tests at 0.01 level are made to ensure the reliability of the results. Linear regression equations are calculated to map the automatic scoring results to human scorings. Introduction Machine translation evaluation has always been a key and open problem. Various evaluation methods exist to answer either of the two questions (Bohan 2000): (1) How can you tell if a machine translation system is “good”? And (2) How can you tell which of two machine translation systems is “better”? Since manual evaluation is time consuming and inconsistent, automatic methods are broadly studied and implemented using different heuristics. Jones (2000) utilises linguistic information such as balance of parse trees, N-grams, semantic co-occurrence and so on as indicators of translation quality. Brew C (1994) compares human rankings and automatic measures to decide the translation quality, whose criteria involve word frequency, POS tagging distribution and other text features. Another type of evaluation method involves comparison of the translation result with human translations. Yokoyama (2001) proposed a two-way MT based evaluation method, which compares output Japanese sentences with the original Japanese sentence for the word identification, the correctness of the modification, the syntactic dependency and the parataxis. Yasuda (2001) evaluates the translation output by measuring the similarity between the translation output and translation answer candidates from a parallel corpus. Akiba (2001) uses multiple edit distances to automatically rank machine translation output by translation examples. Another path of machine translation evaluation is based on test suites. Yu (1993) designs a test suite consisting of sentences with various test points. Guessoum (2001) proposes a semi-automatic evaluation method of the grammatical coverage machine translation systems via a database of unfolded grammatical structures. Koh (2001) describes their test suite constructed on the basis of fine-grained classification of linguistic phenomena. There are many other valuable reports on automatic evaluation. All the evaluation methods show the wisdom of authors in their utilisation of available tools and resources for automatic evaluation tasks. For our localization-oriented lexicalised EBMT system an automatic evaluation module is implemented. Some string similarity criteria are taken as heuristics. Experimental results show that this method is useful in quality feedback in development of the EBMT system. Six machine translation systems are utilised to test the consistency between the automatic method and human evaluation. To avoid stochastic errors, significance test …

Content truncated to 500 words