Scaling the ISLE taxonomy: development of metrics for the multi-dimensional characterization of machine translation quality

Scaling the ISLE Taxonomy: Development of Metrics for the Multi-Dimensional Characterisation of Machine Translation Quality Keith J. Miller, Michelle Vanni The MITRE Corporation US Department of Defense 1820 Dolley Madison Boulevard McLean, VA 22012 USA keith@mitre.org mtvanni@afterlife.ncsc.mil Abstract The DARPA MT evaluations of the early 1990s, along with subsequent work on the MT Scale, and the International Standards for Language Engineering (ISLE) MT Evaluation framework represent two of the principal efforts in Machine Translation Evaluation (MTE) over the past decade. We describe a research program that builds on both of these efforts. This paper focuses on the selection of MT output features suggested in the ISLE framework, as well as the development of metrics for the features to be used in the study. We define each metric and describe the rationale for its development. We also discuss several of the finer points of the evaluation measures that arose as a result of verification of the measures against sample output texts from three machine translation systems. Keywords Machine Translation Evaluation (MTE), ISLE Taxonomy, MT Scale, Task-based evaluation Introduction Attempts at measuring MT output quality have included the comparison of a set of test scores for MT output to a set of the same tests’ scores for naturally-occurring target language text (Jones and Rusk 2000). This work broke new ground in automating MT Evaluation (MTE). However, the tests used were selected on an ad hoc basis and the scores reported on were compared to scores for human-produced text, not necessarily relevant to the text from which the MT was produced. We propose a novel approach to MTE which employs standard, rather than randomly-chosen, features of MT output quality selected from the ISLE framework. This approach also involves a scoring system that has as its goal to predict the type of information processing tasks performable with the output. This methodology is an effort to characterize MT output quality in functional terms while responding to the established desiderata for MTE. These include the capacity to automate and replicate the process as well as to produce results fine-grained enough to be useful to stakeholders such as users, researchers, and developers. Our research program entails a systematic development of the relationship between the evaluation metric (a set of quality test scores) and specific tasks performable on MT output, such as triage, detection, filtering, extraction, and gisting. It is comprised of distinct stages, to include test selection from the ISLE framework, test validation in terms of soundness of design and capacity for replication and automation, approaches to test automation, and the mapping of patterns of test scores to those information- processing tasks performable with the MT output. The issues of score-to-task mapping and validating test selection are crucial to our research program. This paper, however, focuses on the key stage of test development: the selection of MT output features from the ISLE framework and the development of tests to measure system performance with respect to these features, informed by previous approaches. Task-Based MT Evaluation Traditional approaches to …

Content truncated to 500 words