Direct Application of a Language Learner Test to MT Evaluation

Direct Application of A Language Learner Test To MT Evaluation Florence Reeder The MITRE Corporation 7515 Colshire Dr. McLean VA 22102 freeder@mitre.org Abstract This paper shows the applicability of lan- guage testing techniques to machine trans- lation (MT) evaluation through one of a set of related experiments. One straightfor- ward experiment is to use language testing exams and scoring on MT output with little or no adaptation. This paper describes one such experiment, the first in a set. After an initial test (Vanni & Reeder, 2000), we ex- panded the experiment to include multiple raters and a more detailed analysis of the surprising results. Namely that unlike with humans, MT systems perform more poorly at both level zero and one than at level two and three. This paper presents these results as an illustration of both the applicability of language testing techniques and also the caution that needs to be applied. 1 Introduction One basic test that can be performed is the direct application of language learning tests to MT sys- tem outputs. The impetus behind this experiment comes from funding stakeholders, particularly in the United States government. In an effort to bet- ter understand an unfamiliar technology and to quantify what MT can provide, these stakeholders try to frame system evaluations in terms that are familiar to them, specifically, human translator ratings. For such a community, a key question would therefore be: Can scoring criteria for estab- lishing language learner ratings be utilized, as is, for scoring MT output? This experiment was de- signed to contain little or no adaptation for MT. The resulting data set gives indicators of the differ- ences between evaluating language learners and machine translation. The scale used is an Interagency Language Roundtable (ILR) style scale (Lowe & Stansfield, 1988). This is theoretically attractive, as the scale results from years of pedagogical research in un- derstanding and measuring student proficiency in foreign languages. Also, the scoring system is well-known among the intended evaluation con- sumers. The proficiency scale is task-based, so the evaluation can address the needs of multiple stake- holders by answering questions about the functions a student can perform, some of which parallel the uses of MT. From this ILR scale comes a set of teaching recommendations. Because of the de- tailed characteristics described and the test items designed for each level, scoring with this scale permits users to examine relevant linguistic fea- tures that each system provides. Test items are from a ready-made corpus that has been vetted through years of test administration. Finally, since these guidelines and materials are currently being used by the US government, a pool of trained rat- ers exists for multiple languages. 2 Experimental Setup For this experiment, five texts were selected, one for each of the first five ILR levels. The texts were run through three MT systems. Four judges looked at each text in order of increasing difficulty. The sample texts were taken from a course based on the ILR scale. The language pair in …

Content truncated to 500 words