Automatic Evaluation of Machine Translation Based on Rate of Accomplishment of Sub-Goals

Proceedings of NAACL HLT 2007, pages 33–40, Rochester, NY, April 2007. c⃝2007 Association for Computational Linguistics Automatic Evaluation of Machine Translation Based on Rate of Accomplishment of Sub-goals Kiyotaka Uchimoto and Katsunori Kotani and Yujie Zhang and Hitoshi Isahara National Institute of Information and Communications Technology 3-5, Hikari-dai, Seika-cho, Soraku-gun, Kyoto, 619-0289, Japan {uchimoto,yujie,isahara}@nict.go.jp, kat@khn.nict.go.jp Abstract The quality of a sentence translated by a machine translation (MT) system is dif- ficult to evaluate. We propose a method for automatically evaluating the quality of each translation. In general, when translating a given sentence, one or more conditions should be satisfied to maintain a high translation quality. In English- Japanese translation, for example, prepo- sitions and infinitives must be appropri- ately translated. We show several proce- dures that enable evaluating the quality of a translated sentence more appropriately than using conventional methods. The first procedure is constructing a test set where the conditions are assigned to each test-set sentence in the form of yes/no questions. The second procedure is devel- oping a system that determines an answer to each question. The third procedure is combining a measure based on the ques- tions and conventional measures. We also present a method for automatically gener- ating sub-goals in the form of yes/no ques- tions and estimating the rate of accom- plishment of the sub-goals. Promising re- sults are shown. 1 Introduction In machine translation (MT) research, appropriately evaluating the quality of MT results is an important issue. In recent years, many researchers have tried to automatically evaluate the quality of MT and im- prove the performance of automatic MT evaluations (Niessen et al., 2000; Akiba et al., 2001; Papineni et al., 2002; NIST, 2002; Leusch et al., 2003; Turian et al., 2003; Babych and Hartley, 2004; Lin and Och, 2004; Banerjee and Lavie, 2005; Gime´nez et al., 2005) because improving the performance of auto- matic MT evaluation is expected to enable us to use and improve MT systems efficiently. For example, Och reported that the quality of MT results was im- proved by using automatic MT evaluation measures for the parameter tuning of an MT system (Och, 2003). This report shows that the quality of MT re- sults improves as the performance of automatic MT evaluation improves. MT systems can be ranked if a set of MT re- sults for each system and their reference translations are given. Usually, about 300 or more sentences are used to automatically rank MT systems (Koehn, 2004). However, the quality of a sentence translated by an MT system is difficult to evaluate. For exam- ple, the results of five MTs into Japanese of the sen- tence “The percentage of stomach cancer among the workers appears to be the highest for any asbestos workers.” are shown in Table 1. A conventional au- tomatic evaluation method ranks the fifth MT result first although its human subjective evaluation is the lowest. This is because conventional methods are based on the similarity between a translated sentence and its reference translation, and they …

Content truncated to 500 words