Automatic Evaluation of Computer Generated Text: A Progress Report on the TextEval Project

AUTOMATIC EVALUATION OF COMPUTER GENERATED TEXT: A PROGRESS REPORT ON THE TEXTEVAL PROJECT Chris Brew, Henry S. Thompson Human Communication Research Centre University of Edinburgh 2 Buccleuch Place, Edinburgh EH8 9LW, SCOTLAND hthompson@edinburgh.ac.uk ABSTRACT We present results of experiments designed to assess the usefulness of a new technique for the evaluation of translation quality, comparing human rankings with automatic measures. The basis of our approach is the use of a standard set and the adoption of a statistical view of translation quality. This approach has the ability to provide evaluations which avoid dependence on any particular theory of translation, which are therefore potentially more objective than previous techniques. The work presented here was supported by the Science and Engineering and the Social and Economic Research Councils of Great Britain, and would not have been possible without the gracious assistance of Ian Mason of Heriot Watt University, Edinburgh. INTRODUCTION The TextEval project aims to explore and develop a new approach to the automatic evaluation of computer-generated texts, based on the use of standard sets.We believe that fast, accurate and automatic evaluation methods are vital to the development of any large piece of natural language software, and note that current methods, which involve extensive intervention by human experts, are too costly to be a routine part of the development cycle. For pragmatic reasons we are working on translations, but in principle the techniques would apply to any body of suitably comparable texts, such as alternative versions of an instruction sheet. Our purpose in this presentation is to describe a framework for the automatic evaluation of translation quality. We will start by presenting the results of an evaluation experiment, then use the results of this to assess the usefulness of several alternative automatic metrics. TRANSLATION QUALITY. Although it is not clear precisely what translators' subjective judgements are based on when they report on the quality of translations, it is worthy of comment that such judgements can be elicited at all, since attempts to prescribe detailed guidelines for the scientific evaluation of translation quality have been problematic since the ALPAC report [1]. It is arguable that any particular set of guidelines will inappropriately constrain the evaluator by imposing the theoretical preconceptions of the author of the guidelines. In the current project we are exploring the possibility of producing automatic evaluations without any prior commitment to a theory of translation.Our method of evaluation depends only on the essentially statistical hypothesis that good translations will tend to be more similar to each other than will bad ones. Pre-requisites of this approach are the availability of a suitable corpus of translations and the choice of a similarity metric. In this paper the metrics which we describe will be (negative log) probabilities Strictly this makes them dissimilarity metrics. Once we have established a metric, we may apply two approaches to the generation of a rank ordering. In the first approach we find an appropriate method of combining the elements of a set of translations, then measure the distance …

Content truncated to 500 words