Quantitative evaluation of machine translation using two-way MT

MT Summit VII Sept. 1999 Quantitative Evaluation of Machine Translation using Two-way MT Shoichi YOKOYAMA (Yamagata Univ.). Akira KUMANO (Toshiba). Masaki MATSUDAIRA (Oki), Yoshiko SHIROKIZAWA (JST). Mutsumi KAWAGOE (Matsushita). Shuji KODAMA (Fujitsu), Hideki KASHIOKA (ATR), Terumasa EHARA (NHK), Shinichiro MIYAZAWA (Shumei Univ.), and Yasuo NAKAJIMA (AAMT) (Faculty of Engineering. Yamagata University. 4-3-16. Jonan, Yonezawa, Yamagata 992-8510. Japan Tel. +81-238-26-3336. FAX +81-238-26-2082, e-mail: yokoyama®emtsun.yz.yamagata-u.ac.jp) Abstract One of the most important issues in the field of ma- chine translation is evaluation of the translated sen- tences. This paper proposes a quantitative method of evaluation for machine translation systems. The method is as follows. First, an example sentence in Japanese is machine translated into English us- ing several Japanese-English machine translation sys- tems. Second, the output English sentences are ma- chine translated into Japanese using several English- Japanese machine translation systems (different from the Japanese-English machine translation systems). Then, each output Japanese sentence is compared with the original Japanese sentence in terms of word identi- fication, correctness of the modification, syntactic de- pendency, and parataxes. An average score is calcu- lated, and this becomes the total evaluation of the ma- chine translation of the sentence. From this two-way machine translation and the cal- culation of the score, we can quantitatively evaluate the English machine translation. For the present study, we selected 100 Japanese sen- tences from the abstracts of scientific articles. Each of these sentences has an English translation which was performed by a human. Approximately half of these sentences are evaluated and the results are given. In addition, a comparison of human and machine trans- lations is also performed and the trade-off between the two methods of translation is discussed. Keywords: quantitative evaluation, two-way machine translation, word correspondence, modification, comparison of score. 1 Introduction One of the most important issues in the field of ma- chine translation is evaluation of the translated sen- tences. Some of the so-called second generation ma- chine translation systems have adopted and improved on the evaluation method introduced in the ALPAC report. For example, the Mu project, which has devel- oped a Japanese-English machine translation system in Japan, proposed a method of evaluation and defined five degrees of understandability and seven degrees of faithfulness[2]. The degree of understandability is eval- uated by English native speakers and the degree of faithfulness is evaluated by bilingual translators. In their evaluation, the Mu project found that some ex- amples were translated with good understandability. but with bad faithfulness, while other examples were translated with good faithfulness, but with low under- standability. In the Mu project’s evaluation, the highest score of understandability. degree 1. means that “the meaning of sentences is totally clear and undoubted. Grammar, words, and styles are suitable, and therefore no correc- tion is required.” On the other hand, the lowest score of understandability. degree 5. means that “no one can understand the sentences. Even if one thinks over and discusses them, one cannot grasp their meaning.” The highest score of faithfulness, degree 0. means that “the structure …

Content truncated to 500 words