An automatic evaluation method for machine translation using two-way MT

An Automatic Evaluation Method for Machine Translation using Two-way MT Shoichi YOKOYAMA (Yamagata Univ.), Hideki KASHIOKA (ATR), Akira KUMANO (To- shiba), Masaki MATSUDAIRA (Oki), Yoshiko SHIROKIZAWA (JST), Shuji KODAMA (Fu- jitsu), Terumasa EHARA (NHK), Shinichiro MIYAZAWA (Shumei Univ.), and Yuzo MU- RATA (AAMT) Faculty of Engineering, Yamagata University, 4-3-16, Jonan, Yonezawa, Yamagata 992-8510, Japan, yokoyama@emtsun.yz.yamagata-u.ac.jp Abstract Evaluation of machine translation is one of the most important issues in this field. We have already proposed a quantitative evalua- tion of machine translation system. The method was roughly that an example sentence in Japanese is machine translated into English, and then into Japanese using several systems, and that the comparison of output Japanese sentences with the original Japanese sentence is done for the word identification, the correctness of the modification, the syntactic dependency, and the parataxis. By calculating the score, we could quantitatively evaluate the English machine translation. However, the extraction of word identification etc. was done by human, and the fact affects the correctness of evaluation. In order to solve this problem, we developed an automatic evaluation system. We report the detail of the system in this paper.. Keywords: automatic evaluation, two-way machine translation, word correspondence, modification, comparison of score. 1. Introduction Evaluation of machine translation is one of the most im- portant issues in this field. Some of the machine transla- tion systems adopted the human scoring evaluation like the evaluation method of the ALPAC report. This type of evaluation is unstable, and is not quantitative, but qualita- tive. We proposed a quantitative evaluation of machine translation system (Yokoyama, 1999). The method was as follows: first, an example sentence in Japanese is ma- chine translated into English using several Japanese- English machine translation systems. Second, output English sentences are machine translated into Japanese using several English-Japanese machine translation sys- tems (different from Japanese-English machine translation systems). Then, the comparison of output Japanese sen- tences with the original Japanese sentence is done for the word identification, the correctness of the modification, the syntactic dependency, and the parataxis. The average score is calculated based on the proposed method, and it becomes the total evaluation of machine translation for the sentence. From this two-way machine translation and the calcu- lation of the score, we can quantitatively evaluate the English machine translation in which it is difficult to evaluate them for Japanese native speakers. However, most of extraction and comparison processes are performed by human, and the stableness and the speed are problematic. In this paper, we propose an automatized process of the evaluation, and show that the result is the same as by hu- man. In addition, we calculate the score of each Japanese- English machine translation system. The result directly shows the performance of the system. We also try to cal- culate the score of each English-Japanese machine trans- lation system. The result shows that some system makes a good effort even if some Japanese-English translation systems output bad results. Practically, we selected 100 Japanese sentences from abstracts of scientific articles. Each …

Content truncated to 500 words