MT evaluation

MT Summit VII Sept. 1999 MT Evaluation Margaret King ETI/T1M (ex-ISSCO) University of Geneva Eduard Hovy Benjamin K. T'sou USC Information Sciences Institute, City University of Hong Kong USA Hong Kong John White Yusoff Zaharin Litton PRC Universiti Sains Malaysia USA Malaysia Abstract This panel deals with the general topic of evaluation of machine translation systems. The first contribution sets out some recent work on creating standards for the design of evaluations. The second, by Eduard Hovy. takes up the particular issue of how metrics can be differentiated and systematized. Benjamin K. T'sou suggests that whilst men may evaluate machines, machines may also evaluate men. John S. White focuses on the question of the role of the user in evaluation design, and Yusoff Zaharin points out that circumstances and settings may have a major influence on evaluation design. 1 Setting standards for evaluation: Margaret King Evaluation became a critical issue in the world of machine translation at a very early date, with the publication of the ALPAC report in the mid-60's. Highly controversial already at the time of its publication, the report is still capable of rousing strong feelings today. And of course, there is much about it that might be called into question, from the partiality or impartiality of the Committee responsible for the report through to aspects of the evaluation design itself, such as sample size and the validity or invalidity of the metrics chosen. But at least two good things must be said about the ALPAC report. First, it heightened, almost painfully, awareness of the importance of evaluation. Secondly. it did contain a serious attempt to produce a well designed evaluation. Since then, there have been any number of MT evaluations. Indeed, it was once said that probably more had been spent on attempts to evaluate MT than on research and development in the area. Many of these attempts have been laudable contributions to the discipline of evaluation design, and are mentioned as such in the contributions of the panelists. Nonetheless it remains true that there is no single, generally accepted methodology for designing the evaluation of an MT system. In the rest of my remarks, I want to prepare a basis for debate by presenting some recent work, carried out be the Evaluation Working Group of the European EAGLES Initiative (EAGLES. 1999). whose main focus is to propose a framework for designing evaluations, not just of MT systems but of human language technology systems in general. The work is closely related to two ISO standards. ISO/IEC 9126 and ISO/IEC 14598. which are both concerned with the even more general question of evaluation of software product. The basic thrust behind both the ISO work and the EAGLES specialization of it to HLT systems is not to produce evaluations for specific classes of systems worked out to the last detail, but to provide standards and guidelines for constructing specific evaluations. The value of doing so is manifold: the standards and guidelines provide the evaluator with a readily available …

Content truncated to 500 words