Task-based evaluation for machine translation

MT Summit VII Sept. 1999 Task-Based Evaluation for Machine Translation Jennifer B. Doyon, John S. White Kathryn B. Taylor Litton PRC QSI Inc. 1500 PRC Drive 8201 Greensboro Drive McLean, VA 22102 Suite 1200 (doyon_jennifer, McLean, VA 22102 white_john}@prc.com taylorkb@erols.com Abstract In an effort to reduce the subjectivity, cost, and complexity of evaluation methods for machine translation (MT) and other language technologies, task-based assessment is examined as an alternative to metrics-based in human judgments about MT, i.e., the previously applied adequacy, fluency, and informativeness measures. For task-based evaluation strategies to be employed effectively to evaluate language- processing technologies in general, certain key elements must be known. Most importantly, the objectives the technology’s use is expected to accomplish must be known, the objectives must be expressed as tasks that accomplish the objectives, and then successful outcomes defined for the tasks. For MT, task-based evaluation is correlated to a scale of tasks, and has as its premise that certain tasks are more forgiving of errors than others. In other words, a poor translation may suffice to determine the general topic of a text, but may not permit accurate identification of participants or the specific event. The ordering of tasks according to their tolerance for errors, as determined by actual task outcomes provided in this paper, is the basis of a scale and repeatable process by which to measure MT systems that has advantages over previous methods. 1 Introduction It is by now well known that machine translation (MT) evaluation is significantly different from evaluation of other language processing technologies, because of the fact there is no single “right” translation of any expression, and thus no single ground truth for comparison. In general, the techniques devised to mitigate this difficulty have involve: using target-native speakers to make judgments about the fidelity and intelligibility of MT output (White et al. 1994; White 1995; Doyon et al. 1998). These methods have a disadvantage of requiring large numbers of judgments, texts, and raters to control against biasing effects. Meanwhile, the context of MT (and all language processing) has shifted in the last decade from the presumption of turnkey, single function systems to end-to- end production environments that integrate multiple automatic language processing systems into a single process flow. While the intelligibility and fidelity measures remain valuable in comparing MT systems, they do not directly indicate the contribution (or degradation that an MT system makes in the context of other processes such as topic detection, information extraction, gisting, summarization, and so on. New evaluation methods must be able to take optimum advantage of human judgments without the heavy resource requirements, and at the same time measure something useful for the new context of operation. While an evaluation strategy is now a required part of the development of any technology, there are few standards available which measure either the progress or usefulness of any individual MT system. Self-reporting by system developers is not always a cost-effective or reliable method to measure the progress of a research project and …

Content truncated to 500 words