Approaches to black box MT evaluation

Approaches to Black Box MT Evaluation John S. White PRC Inc. 1500 PRC Drive McLean, VA 22102 USA white_john@prc.com Abstract. In the course of four evaluations in the Advanced Research Projects Agency Machine Translation series, evaluation methods have evolved for measuring the core components of a diverse set of systems. This paper describes the methodologies in terms of the most recent evaluation of research and production MT systems, and discusses indications of ways to improve the focus and portability of the evaluation. 0. Introduction. Over the past four years, a set of evaluation methodologies have evolved within the MT initiative of the U.S. Advanced Research Projects Agency (ARPA). The ARPA program has faced unique challenges for evalua- tion, because the systems partic- ipating differ radically in the linguistic approach, their level of maturity, and the languages translated. The differences among these systems have made a black-box orientation to evaluation in- evitable. While such an orien- tation differs from the methods that might be employed in the evaluation of a particular sys- tem by that system's developers, there are nevertheless certain advantages to the black box ap- proaches in determining the fo- cus and metrics of evaluation. This paper describes the methodologies of the ARPA program, in terms of their objec- tives, results, and evolution, and discusses analyses of the methods themselves to con- tinue to improve the process. 1.0. Background. The ARPA initiative in machine transla- tion began in 1991 as part of the Human Language Technologies Program. Three projects in MT research were sponsored under the initiative, with voluntary participation by several com- mercial and institutional MT organizations. The sponsored projects were: Candide (IBM Watson Research Center), a sta- tistical modeling approach, translating French to English (Brown et al., 1993); Pangloss (Center for Machine Translation, Computing Research Laboratory, and Institute for Information Science), using knowledge- based approaches, translating Spanish and Japanese to English (Frederking et al., 1993); and Lingstat (Dragon Systems, Inc.), using a combination of model- ing and rule-based approaches, translating Japanese and Spanish to English (Yamron et al., 1994). Many organizations have pro- vided production MT systems for these evaluations, princi- pally to assist ARPA's mission by helping to determine indus- try/discipline benchmarks. A significant goal of the ARPA MT Evaluation program, in re- turn, is to provide a useful set of evaluation processes for a gen- eral standard. In the most re- cent test-evaluation cycle of August - November 1994 (the "3Q94" evaluation), the follow- ing production systems partici- pated: • the Sietec METAL system (French - English); • the Nippon Electric PIVOT system (Japanese - English); • Globalink Power Translator (French and Spanish - English); • the Pan American Health Organization SPANAM system (Spanish - English); • Systran (French, Japanese, and Spanish - English); the SOCATRA XLT sys- tem (French - English). 2.0. Testing. ARPA's mission and focus has been toward ad- vancement of the "core" tech- nologies underlying MT. Even though it is fully recognized that all MT systems in the fore- seeable future …

Content truncated to 500 words