QARLA: A Framework for the Evaluation of Text Summarization Systems

Proceedings of the 43rd Annual Meeting of the ACL, pages 280–289, Ann Arbor, June 2005. c⃝2005 Association for Computational Linguistics QARLA:A Framework for the Evaluation of Text Summarization Systems Enrique Amig´o, Julio Gonzalo, Anselmo Pe˜nas, Felisa Verdejo Departamento de Lenguajes y Sistemas Inform´aticos Universidad Nacional de Educaci´on a Distancia c/Juan del Rosal, 16 - 28040 Madrid - Spain {enrique,julio,anselmo,felisa}@lsi.uned.es Abstract This paper presents a probabilistic framework, QARLA, for the evaluation of text summarisation systems. The in- put of the framework is a set of man- ual (reference) summaries, a set of base- line (automatic) summaries and a set of similarity metrics between summaries. It provides i) a measure to evaluate the quality of any set of similarity metrics, ii) a measure to evaluate the quality of a summary using an optimal set of simi- larity metrics, and iii) a measure to eval- uate whether the set of baseline sum- maries is reliable or may produce biased results. Compared to previous approaches, our framework is able to combine different metrics and evaluate the quality of a set of metrics without any a-priori weight- ing of their relative importance. We pro- vide quantitative evidence about the ef- fectiveness of the approach to improve the automatic evaluation of text sum- marisation systems by combining sev- eral similarity metrics. 1 Introduction The quality of an automatic summary can be es- tablished mainly with two approaches: Human assessments: The output of a number of summarisation systems is compared by hu- man judges, using some set of evaluation guidelines. Proximity to a gold standard: The best auto- matic summary is the one that is closest to some reference summary made by humans. Using human assessments has some clear ad- vantages: the results of the evaluation are inter- pretable, and we can trace what a system is do- ing well, and what is doing poorly. But it also has a couple of serious drawbacks: i) different hu- man assessors reach different conclusions, and ii) the outcome of a comparative evaluation exercise is not directly reusable for new techniques, i.e., a summarisation strategy developed after the com- parative exercise cannot be evaluated without ad- ditional human assessments made from scratch. Proximity to a gold standard, on the other hand, is a criterion that can be automated (see Section 6), with the advantages of i) being objective, and ii) once gold standard summaries are built for a com- parative evaluation of systems, the resulting test- bed can iteratively be used to refine text summari- sation techniques and re-evaluate them automati- cally. This second approach, however, requires solv- ing a number of non-trivial issues. For instance, (i) How can we know whether an evaluation met- ric is good enough for automatic evaluation?, (ii) different users produce different summaries, all of them equally good as gold standards, (iii) if we have several metrics which test different features of a summary, how can we combine them into an optimal test?, (iv) how do we know if our test bed 280 Figure 1: …

Content truncated to 500 words