Evaluation experiment for reading comprehension of machine translation outputs

MT Summit VII Sept. 1999 Evaluation Experiment for Reading Comprehension of Machine Translation Outputs Masaru Fuji Document Processing Lab Fujitsu Laboratories Ltd. Abstract This paper proposes evaluation methods for reading comprehension of English to Japanese translation outputs. The meth- ods were designed not only to evaluate the performance of current systems, but to eval- uate the performance of future systems had the current problems been solved. The experiments have shown that the pro- posed methods are capable of producing results that are statistically significant, and that improvement in certain linguistic as- pects will result in significant improvement in the comprehension level. 1 Introduction In recent years, machine translation (MT) has cre- ated a large market in Japan. Currently, the mam users of commercial MT systems are those who wish to browse through information expressed in a foreign lan- guage, and NOT professional translators whose task is to produce complete translation documents. Those users who wish to seek information written in a for- eign language will be interested to know how much information can be retrieved from the translated text and how easily it is retrieved. Under these circumstances, the Japanese MT providers are interested to find out. on what sort of linguistic as- pects they need to concentrate in their development processes, in order to improve browsability. They are also interested to know how much improvement can be expected from the development effort they put in. This is exactly the motivation behind our research. 1.1 Related works We have classified traditional MT evaluation methods into the following three categories. Since our target is to devise evaluation methods for prospective end users, we have adopted the last category - usability test - for our experiments. • Evaluation using test sets Test set is a set of sample sentences with their ideal translation sentences [3]. The sample sen- tences are fed into the MT system and the out- put compared to the ideal translations. This is probably the most popular test for MT de- velopers to monitor their system development, though the test sets are usually used within their private development environment and much of the results are unpublished to academic confer- ences. It offers systematic evaluation and gives measurable evaluation results. The results are unlikely to represent the comprehension level of end users. • Inspection test by trained evaluators MT output sentences are judged by linguists or MT developers and scores given, such as "good", "poor" , etc. This is a quick and easy way of esti- mating the translation quality produced by the MT system. Some researchers have pointed out that, in inspection test, there is a number of in- dependent measurable aspects such as informa- tiveness, comprehensiveness, etc. However, the result may not exactly correspond to the impres- sion received by general users. • Usability test The target MT users are used as the subjects of evaluation. The results obtained closely cor- respond to the general users' impression. How- ever, it is necessary to use a large number of …

Content truncated to 500 words