Trial and error: an evaluation project on Japanese <> English MT output quality

Trial and Error: An Evaluation Project on Japanese <> English MT Output Quality Maki Darwin Mendez, Inc. 5095 Murphy Canyon Rd., #300, San Diego, CA 92123 U.S.A. Maki.Darwin@lhsl.com Abstract This paper describes a small-scale but organized attempt to evaluate output quality of several Japanese MT systems. The project also served as the first experiment of the implementation of the in-house MT evaluation guidelines created in 2000. Since time was limited and the budget was not infinite, it was launched with the following compact components: Five people; 300 source sentences per language pair; and 160 hours per evaluator. The quantitative results showed noteworthy phenomena. Although the test materials had been presented in a way that evaluators could not identify the performance of any particular system, the results were quite consistent. The scoring ratio that the two E-to-J evaluators employed was almost identical, while that of the J-to-E evaluators was similar. This indicates that high-quality output has universal appeal. Additionally, the evaluators noted that stronger systems, regardless of language pair, tended to be superior in source sentence analysis, target sentence arrangement, word choice, and lexicon entries whereas weaker systems tended to be inferior in these areas. As for language-pair comparison, the results indicate that English-to-Japanese systems may require more improvement than their counterparts, judging from the scores given and the number of unfound words recorded. Keywords MT evaluation techniques and results; A case study of MT evaluation. Part 1: Evaluation Environment 1. Evaluation Goals This project aimed to evaluate the translation quality of leading machine translation systems on the Japanese market by following the in-house evaluation guidelines created in June 2000. The language pairs tested include English-to-Japanese and Japanese-to-English. Another purpose was to assess the translation quality of L&H’s Japanese-English MT systems in comparison with competitors’ products. The evaluation team consisted of the following five people: Coordinator (Maki Darwin); two E-to-J Evaluators, both native speakers of Japanese (Evaluator A and Evaluator B); and two J-to-E Evaluators, including one native speaker of English (Evaluator C) and one equivalent of English speaker, a native speaker of Japanese (Evaluator D). All of them have experience in either E-to-J or J-to-E translation as freelance translators or business personnel. Each evaluator was assigned 160 hours to complete his/her evaluation. 2. MT Systems Evaluated The project evaluated eight systems for each language pair. They are marked as [EJsys-1]…[EJsys-8] for the E- to-J direction and as [JEsys-1]…[JEsys-8] for the J-to-E direction. Most of the products tested were bi-directional, each direction sharing the same system number (e.g.: [EJsys-8] and [JEsys-8]); in other cases, mono-directional products from the same maker shared the same system number. To attain the greatest possible objectivity, the evaluators were not shown the identities of these products throughout the project; all the systems were shown as code names only, both when the evaluators scored the output and analyzed their evaluation results. As a rule, the systems were tested in “as is” condition. Therefore, the default settings for their translation output remained unless a setting contradicted the purpose …

Content truncated to 500 words