A test suite for evaluation of English-to-Korean machine translation systems

A Test Suite for Evaluation of English-to-Korean Machine Translation Systems Sungryong Koh, Jinee Maeng, Ji-Young Lee, Young-Sook Chae, Key-Sun Choi Korea Terminology Research Center for Language and Knowledge Engineering (KORTERM) Korea Advanced Institute of Science and Technology (KAIST) Kusong-dong Yusong-gu Taejon 305-701 Korea {koh,aphroditejin,jinny206}@world.kaist.ac.kr, pinochae@chollian.net, kschoi@cs.kaist.ac.kr Abstract This paper describes KORTERM’s test suite and their practicability. The test-sets have been being constructed on the basis of fine- grained classification of linguistic phenomena to evaluate the technical status of English-to-Korean MT systems systematically. They consist of about 5000 test-sets and are growing. Each test-set contains an English sentence, a model Korean translation, a linguistic phenomenon category, and a yes/no question about the linguistic phenomenon. Two commercial systems were evaluated with a yes/no test of prepared questions. Total accuracy rates of the two systems were different (50% vs. 66%). In addition, a comprehension test was carried out. We found that one system was more comprehensible than the other system. These results seem to show that our test suite is practicable. Keywords Evaluation, English-to-Korean, Yes-No Question, Comprehension Test, Linguistic Phenomena 1. Introduction It has been emphasized that we have to evaluate the quality of translation from the specific purpose of an evaluation. The purpose of an evaluation is generally related to who a user is and what a task is. For example, a manager may want to read a letter or an email from foreign employees using an MT system. A developer of an MT system wants to test the performance of specific processes. The explicit description of the purpose helps to identify what characteristics of translation of a MT system should be measured. Since a manager or an end-user is usually interested in a general performance of a system, the degree of comprehensibility or fidelity of translation could be measured using a questionnaire. On the other hand, errors of a system about a variety of linguistic phenomena could be measured, since a system developer is interested in whether specific processes of translation have a problem. JEIDA (1992) showed a neat method that relates users’ needs to MT systems. In order to help the user evaluation of economic factors, several questions about the conditions of translation work and the user’s needs are prepared. Their answers are analyzed into 14 parameters such as present translation needs, type of document and so on. The analysis result is represented as a radar chart and is compared with radar charts that characterize seven groups of MT systems. This comparison makes it possible to identify a system close to the user’s needs. Recently the importance of the usability of a product has been recognized and stressed. EAGLES Evaluation working group (1999) proposed a general framework for evaluation following ISO quality model. Their report emphasized the importance of quality in use as well as quality of a product. They defined quality in use as the user’s view of the quality of a system containing software, claiming that it is measured in terms of the results of the use of …

Content truncated to 500 words