JEIDA’s test - sets for quality evaluation of MT systems

JEIDA's Test-Sets for Quality Evaluation of MT Systems -- Technical Evaluation from the Developer's Point of View -- Hitoshi ISAHARA Kansai Advanced Research Center Communications Research Laboratory 588-2, Iwaoka, Iwaoka-cho, Nishi-ku, Kobe, 651-24, JAPAN E-mail. isahara@crl.go.jp abstract This paper describes a method of evaluating quality for developers of machine translation systems to easily check imperfections in their own systems. This evaluation method is a systematic, objective method along with test example sets in which we clarified the evaluation procedure by adding yes/no questions and explanations to the example sentences for evaluation. In March 1995, our two test-sets (English-to-Japanese and Japanese-to-English) were completed and made publicly available. 1. Introduction Since 1992, we have been developing a method of evaluating quality for the developers of machine translation (MT) systems to allow them to easily check imperfections in their own systems [1, 2, 3, 4, and 5]. In March 1994, we made our first version of the test-sets which evaluate the quality of English-to-Japanese Machine Translation systems (the "1993 Test Sets") available to the public. This test-sets is being used by developers, researchers and users of machine translation systems inside and outside Japan. We have continued our efforts to solve the existing problems revealed by evaluation experiments on some commercial MT systems and to increase the number of example sentences so as to cover more linguistic phenomena. Also, we have started to develop our test-sets for Japanese-to- English machine translation systems. In March 1995, our two test-sets (English-to-Japanese and Japanese-to-English) were completed and made publicly available. In this paper, we would like to describe this systematic, objective method along with the test sets in which we have clarified the evaluation procedure by adding questions and explanations to the examples for the evaluation. The work described in this paper has been developed by the Special Interest Group on Machine Translation (Chief: Hitoshi ISAHARA) in the Natural Language Processing System Research Committee (Chairman: Prof. Hozumi TANAKA, Tokyo Institute of Technology) which is a subcommittee of the Natural Language Processing Technology Committee (Chairman: Prof. Makoto NAGAO, Kyoto University) of JEIDA (Japan Electronic Industry Development Association). The members of the Special Interest Group in 1994 are shown in Table 1. JEIDA has formulated three criteria for evaluating MT systems: 1) technical and 2) financial evaluations for users, and 3) technical evaluation for developers. For more information on these criteria, please refer to references 1 and 6. Table 1. Members of the Special Interest Group Hitoshi ISAHARA (Communications Research Laboratory, MPT) Hajime UCHINO (Nippon Telegraph and Telephone) Shiho OGINO (Ninon IBM) Toshiyuki OKUNISHI (Sharp Corp.) Satoshi KINOSHITA (Toshiba) Shogo SHIBATA (CANON INC.) Toshiyuki SUGIO (Oki Electric Industry Co., Ltd.) Yasuhiro TAKAYAMA (Mitsubishi Electric Corp.) Shin'ichi DOI (NEC Corp.) Tadashi NAGANO (Matsushita Electric Industrial Co., Ltd.) Masumi NARITA (Ricoh Co., Ltd.) Hirosato NOMURA (Kyushu Institute of Technology) We will first describe how our evaluation method surpasses previous methods, with reference to the following 2 types of objectivity. (1) Objectivity in the evaluation process (2) Objectivity in the judgment of …

Content truncated to 500 words