Towards an automated evaluation of an embedded MT system

106 Towards an Automated Evaluation of an Embedded MT System J. Laoudi#,* #ARTI Alexandria, Virginia jlaoudi@arl.army.mil C. Tate±,* ±Dept. of Mathematics U. of Maryland, College Park, Maryland ctate@math.umd.edu C. R. Voss* *Multilingual Computing Group Army Research Lab Adelphi, Maryland voss@arl.army.mil Abstract. How can recent advances in automating the evaluation of machine translation (MT) engines be applied to automate the evaluation of more complex embedded MT systems? In this paper, we describe initial evaluation testing of FALCon, an embedded MT system where hard-copy documents are scanned into bit-mapped images, converted into online text files via optical character recognition (OCR) software, and then translated by an MT engine from one natural language into another. Our challenge with the ongoing support of FALCon systems is to evaluate when the replacement of a particular component with a new or upgraded product will yield significant improvements over the baseline system performance. In this paper we address the following questions: (i) how can we automate the baseline end-to-end evaluation of this system? and (ii) what is the relation between the accuracy of the individual components and their end-to-end accuracy, that could be used to model the system performance? 1. Introduction How can recent advances in automating the evaluation of machine translation (MT) engines1 be applied to automate the evaluation of more complex embedded MT systems2? Consider, for example, FALCon, an embedded system whose process flow appears in Figure 1. First the hard-copy document pages are scanned into bit-mapped images, then converted into online text files via optical character recognition (OCR) software, and then translated by an MT engine from one natural language into another [3,4]. Our challenge with the ongoing support of FALCon systems in the field is to evaluate when the replacement of a particular component with a new or upgraded product will yield significant improvements in system performance. In this paper we address the following questions: (i) how can we automate the baseline end-to-end evaluation of this system? and (ii) what is the relation between the accuracy of the individual components and their end-to-end 1 See [2],[7],[8],[9] 2 Embedded MT systems are defined as computational systems with one or more MT engines embedded as modules in the process flow of the system [15,16,17]. accuracy, that could be used to model the system performance, to assess the added value of engineering new components into the system? Our work in evaluating FALCon has led us to pursue both the task-based approach of testing users in hands-on experiments [13] and the text-based approach of assessing the system output against human reference translations [14]. The former yields measures of effectiveness that provide our users with an estimate of their level of accuracy on specific task applications using this system, while the latter yields measures of performance that capture different aspects of the system’s lexical, morphological and grammatical coverage. A full system evaluation ultimately requires both types of measures and determining the relation between them.3 3 The relation between these two types of measures is discussed in [12]. The …

Content truncated to 500 words