Evaluation method for determining groups of users who find MT “useful”

Evaluation Method for Determining Groups of Users Who Find MT “Useful” M. Fuji (Fujitsu Labs), N. Hatanaka (Canon), E. Ito (Toshiba), S. Kamei (NEC), H. Kumai (Hitachi), T. Sukehiro (Oki), T. Yoshimi (Sharp), H. Isahara (Communications Research Lab) Fujitsu Laboratories Ltd, 4-1-1, Kamikodanaka, Nakahara, Kawasaki, Kanagawa 211-8588, JAPAN. fuji.masaru@jp.fujitsu.com Abstract This paper describes an evaluation experiment designed to determine groups of subjects who prefer reading MT outputs to reading the original text. Our approach can be applied to any language pairs, but we will explain the methodology by taking English to Japanese translation as an example. In the case of E-J MT, it can be assumed that main users are Japanese and that most of them have some knowledge of English. It is often the case, in the case of E-J MT systems, that those people who are comfortable with reading English do not find E-J MT outputs useful, and in many cases, they would rather prefer reading the original English text. On the other hand, E- J MT outputs prove to be useful to those who find it hard to read the original English texts. We have used the reading comprehension part of the Test Of English for International Communication (TOEIC) to determine the threshold English ability level, dividing these two user groups. Keywords English-Japanese MT, evaluation, reading comprehension, TOEIC and language ability. Research body This evaluation experiment has been designed and carried out as part of the activity for the Technical Research Committee of the Asia-Pacific Association for Machine Translation (AAMT). Background A large number of E-J MT products are currently put on market in Japan. One of the main purposes of E-J MT systems is to enable Japanese users to browse, in their own language, through vast quantities of English documents. Although the output quality of MT systems has been improved recently through continuous research and development effort, the quality is not high enough to satisfy all the potential users. The reality is that those Japanese people who have sufficient command of English would prefer reading original English texts, rather than reading the corresponding Japanese text translated using MT. On the contrary, there are a certain number of people, who are less confident with reading English and who would much prefer reading MT translated Japanese text, even though the output quality is nowhere near perfection. Objective The objective of our research work is to devise an experiment for evaluating the quality of MT systems in the form that is easily understood by the general public. The experiment is designed in such a way as to determine the groups of subjects who find MT outputs useful. The methodology we adopt here should be such that it can be applied to any language pairs, though we take English to Japanese translation as an example. General idea We make use of the TOEIC as a convenient scale to measure MT quality. As large numbers of people take this test world wide, it is hoped that the scores obtained using this …

Content truncated to 500 words