Technical Evaluation of MT Systems from the Developer’s Point of View: Exploiting Test-Sets for Quality Evaluation

Technical Evaluation of MT Systems from the Developer's Point of View: Exploiting Test-Sets for Quality Evaluation Isahara, H.*(Electrotechnical Laboratory, MITI), Uchino, H. (Nippon Telegraph and Telephone), Ogino, S. (Nihon IBM), Okunishi, T. (Sharp Corp.), Kinoshita, S. (Toshiba), Shibata, S. (CANON INC.), Sugio, T. (Oki Electric Industry Co., Ltd.), Takayama, Y. (Mitsubishi Electric Corp.), Doi, S. (NEC Corp.), Nagano, T. (Matsushita Electric Industrial Co., Ltd.), Narita, M. (Ricoh Co., Ltd.), and Nomura, H. (Kyushu Institute of Technology) Abstract This paper describes a method of evaluating quality for developers of machine translation sys- tems to easily check imperfections in their own systems. This evaluation method is a systematic, objective method along with test example sets in which we clarified the evaluation procedure by adding yes/no questions and explanations to the example sentences for evaluation. 1 Introduction Since 1992, we have been developing a method of evaluating quality for the developers of machine translation (MT) systems to easily check imperfections in their own systems [1, 2, and 3]. In this paper, we would like to describe this systematic, objective method along with the test example sets in which we have clarified the evaluation procedure by adding questions and explanations to the examples for the evaluation1. We will first describe how our evaluation method surpasses previous methods, with reference to the following 2 types of objectivity. (1) Objectivity in the evaluation process (2) Objectivity in judgment of the evaluation results *isahara@etl.go.jp 1 The work described in this paper is being developed by the Special Interest Group on Machine Translation (Chief: Hitoshi ISAHARA, Electrotechnical Laboratory) in the Natural Language Processing System Research Committee (Chairman: Prof. Hozumi TANAKA, Tokyo Institute of Technology) which is a subcommittee of the Natural Language Processing Technology Committee (Chairman: Prof. Makoto NAGAO, Kyoto University) of JEIDA (Japan Electronic Industry Development Association). JEIDA has formulated three criteria for evaluating MT systems: 1) technical and 2) economical evaluations for system users, and 3) technical evaluation for the system developers. For more information on these criteria, please refer to references 1 and 4. 126 In an evaluation method such as the one proposed in the ALPAC report, "fidelity" and "intel- ligibility" are employed as evaluation measures, though they are dependent on human, subjective judgment. Consequently, the results may differ according to who has made the evaluations, that is, they do not satisfy the objectivity criterion (1). Theoretically, the evaluation method in the ALPAC report satisfies criterion (2) since the evaluation results are given as numbers. The system developers, however, fail to recognize which items cannot be handled in their own system. This is because the test example in question covers various kinds of grammatical items. So, their inter- pretation of the evaluation result for further improvement of their system must still be subjective. Therefore, for all practical purposes, this evaluation method does not satisfy criterion (2). On the other hand, we have been preparing test-sets that can satisfy both objectivity criteria (1) and (2). There, we have clarified how to evaluate individual examples posing …

Content truncated to 500 words