Evaluation Metrics for Knowledge-Based Machine Translation

Evahmtion Metrics t'oi- Knowledge-Based Machine Translation Eric I1. Nyberg, 3rd Teruko Mitamura Jaimc G. Carbonell Ccnte, r for Machine Translation Carnegie Mellon University Pittsburgh, PA 15213 Topical Paper: machine translation Abstract A methodology is presented for coml)onent-l)ase(l machine translation (MT) evaluation through causal error analysis to complement existing global evalu- ation methods. This methodology is particularly :q)- propriate for knowledgc-I)ased machine translation (KBMT) systems. After a discussion o[ M'I' eval- uation criteria and the particular evahlatiou metrics proposed for KBMT, we apply this methodology to a large-scale application of the KANT ,nachinc translation system, and present some sample results. 1 Introduction Machine Translation (MT) is considered the paradigm task of Natural Language Processing (NLP) hy some researchers because it combines almost all NLP research :treas: syntactic parsing, semantic disambigt, ation, knowledge rel)reseutation, language generation, lexical acquisition, and morphological analysis and synthesis. However, the evaluation method- ologies for MT systems have heretofore centered on hlack box approaches, where global properties of tile system are evaluated, such as semantic fidelity of the translation or com- prehensibility of the target langt,age output. There is a long tradition of such black-box MT evaluations (Van Slype, 1979; Nagao, 1985; JEIDA, 1989; Wilks, 1991), to the point that Yorick Wilks has stated: "MT Evaluation is better understood than MT" (Carbonell&Wilks. 1991 ). While these evalt,,'ltions are extremely important, they should be augmented with de- tailed error analyses and with component cval uation s in ordcr to produce causal analyses l)inpointing errors and therefm'e leading to system improvement. In essence, we advocate both causal component analyses as well as gloi)al behavioral anal- yses, preferably when the latter is consistent with tile Iormer via composition of the component analyses. Tim advent of Knowledge Based Machine Translation (KBMT) facilitates component evaluation and error attribu- tion because of its modular nature, though this ol)servalion by no means excludes transfer-based systems from similar aualyses. After reviewing the reasons att(I criteria for MT evaluation, this paper describes a specific evaluation method- ology and its application to the KANT system, developed at CMU's Center for Machine Translation (Mitamura, et al. 1991). The KANT KBMT architecture is particularly well- suited for detailed evaluation because of its relative simplicity ':ompared to other KBMT systems, and because it has been scaled up to industrial-sized al)plications. 2 Reasous for Evaluation Machine Translation is evaluated for a number of different reqsons, and when possihle these should be kept clear and separate, as diflerent types of ev,'duation are best suited to measure different aspects of an MT system, l.et ns review the reasons wily MT systems may be evaluated: • Com/)arison with llumans. It is useltd to establish a global comparison with hurmm-qu:.dity translation as a function of task. For general-ptnl)OSe accurate tralls- lation, most MT systelns have a long way to go. A behavioral black-box evahmtion is appropriate here. ® Decision to use or buy a particular MT syMet~.t. This evahmliou is task dependent, aud nmst take both quality of trallslation as well as economics inR) accf)nllt (e.g. …

Content truncated to 500 words