Feedback Cleaning of Machine Translation Rules Using Automatic Evaluation

Feedback Cleaning of Machine Translation Rules Using Automatic Evaluation Kenji Imamura, Eiichiro Sumita ATR Spoken Language Translation Research Laboratories Seika-cho, Soraku-gun, Kyoto, Japan {kenji.imamura,eiichiro.sumita}@atr.co.jp Yuji Matsumoto Nara Institute of Science and Technology Ikoma-shi, Nara, Japan matsu@is.aist-nara.ac.jp Abstract When rules of transfer-based machine translation (MT) are automatically ac- quired from bilingual corpora, incor- rect/redundant rules are generated due to acquisition errors or translation variety in the corpora. As a new countermeasure to this problem, we propose a feedback cleaning method using automatic evalua- tion of MT quality, which removes incor- rect/redundant rules as a way to increase the evaluation score. BLEU is utilized for the automatic evaluation. The hill- climbing algorithm, which involves fea- tures of this task, is applied to searching for the optimal combination of rules. Our experiments show that the MT quality im- proves by 10% in test sentences according to a subjective evaluation. This is consid- erable improvement over previous meth- ods. 1 Introduction Along with the efforts made in accumulating bilin- gual corpora for many language pairs, quite a few machine translation (MT) systems that automati- cally acquire their knowledge from corpora have been proposed. However, knowledge for transfer- based MT acquired from corpora contains many in- correct/redundant rules due to acquisition errors or translation variety in the corpora. Such rules con- flict with other existing rules and cause implausible MT results or increase ambiguity. If incorrect rules could be avoided, MT quality would necessarily im- prove. There are two approaches to overcoming incor- rect/redundant rules: • Selecting appropriate rules in a disambiguation process during the translation (on-line process- ing, (Meyers et al., 2000)). • Cleaning incorrect/redundant rules after automatic acquisition (off-line processing, (Menezes and Richardson, 2001; Imamura, 2002)). We employ the second approach in this paper. The cutoff by frequency (Menezes and Richardson, 2001) and the hypothesis test (Imamura, 2002) have been applied to clean the rules. The cutoff by fre- quency can slightly improve MT quality, but the im- provement is still insufficient from the viewpoint of the large number of redundant rules. The hypothesis test requires very large corpora in order to obtain a sufficient number of rules that are statistically confi- dent. Another current topic of machine translation is automatic evaluation of MT quality (Papineni et al., 2002; Yasuda et al., 2001; Akiba et al., 2001). These methods aim to replace subjective evaluation in or- der to speed up the development cycle of MT sys- tems. However, they can be utilized not only as de- velopers’ aids but also for automatic tuning of MT systems (Su et al., 1992). anWe propose feedback cleaning that utilizes automatic evaluation for removing incor- rect/redundant translation rules as a tuning method Training Corpus Automatic Acquisition Translation Rules Evaluation Corpus MT Engine Automatic Evaluation MT Results Rule Selection/Deletion Feedback Cleaning Figure 1: Structure of Feedback Cleaning (Figure 1). Our method evaluates the contribution of each rule to the MT results and removes inap- propriate rules as a way to increase the evaluation scores. Since the automatic evaluation …

Content truncated to 500 words