Adaptation of the DARPA machine translation evlauation paradigm to end-to-end systems

ADAPTATION OF THE DARPA MACHINE TRANSLATION EVALUATION PARADIGM TO END-TO-END SYSTEMS John S. White Theresa A. O'Connell Research and Development Litton PRC 1500 PRC Drive McLean, VA 22102-5050 tel: 703-556-1899 Abstract The Defense Advanced Research Projects Agency (DARPA) Machine Translation (MT) Initiative spanned four years. One outcome of this effort was a methodology for evaluating the core technology of MT systems which differ widely in approach, maturity, platform, and language combination. This black box methodology, which proved capable of measuring performance of such diverse systems, used methods which collected human judgments on MT output; expressed these judgments in a quantitative form; and assessed them to evaluate the quality of MT output. This paper investigates the feasibility of applying this methodology to MT efforts beyond the original DARPA goals, into evaluation of MT as part of end-to-end multilingual document understanding systems. 0. Introduction The Advanced Research Projects Agency (DARPA) Machine Translation (MT) Initiative was a four-year, comprehensive effort intended to develop new MT approaches and translation algorithms. In addition, the initiative developed methodologies for evaluating MT systems, which were appropriate for measuring the progress of core translation algorithms (White et al., 1994). Today, practical application of MT in the workplace introduces new challenges to MT evaluation. Work currently underway at the Federal Intelligent Document Understanding (IDU) Laboratory has identified the need for innovative approaches to evaluate MT systems which function as components of larger end-to-end systems (Craig, 1995). This paper describes some of the challenges this effort faces. It offers modifications to the DARPA MT Evaluation Methodology which speak to the wider needs of applied MT. The ultimate functionality of MT will hinge on its ability to operate within systems performing multi-step text handling operations, such as optical character recognition (OCR), detection, and extraction. The role of MT in these systems will be quite different from the conventional view of MT as a standalone system. The burdens added to the MT system include: interoperability with other components of the end-to-end system; quality of the output of components which 106 feed into the MT component; impact of the quality of MT output on other system components; and human factors issues which bear upon human computer interaction. The principles of the DARPA MT Evaluation methodology may be applied to MT components in this context, and appear, with some modification, to be germane to several of the end-to-end text handling tasks. 1. The DARPA Methodology A significant goal of the DARPA MT Evaluation program was to develop a set of evaluation processes for a general standard. The methodology developed is based on human judgments, taking advantage of the highly subjective nature of opinions about the accuracy of all translation (including human expert). The DARPA solution to handling the subjectivity of human assessments was to decompose these judgments into a large sample of small units, focused and controlled by separate evaluations for adequacy, informativeness, and fluency (White et al., 1995). Expert human translations served as ground truth. This methodology successfully assessed the quality of the …

Content truncated to 500 words