Mind the Gap: Dangers of Divorcing Evaluations of Summary Content from Linguistic Quality

Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008), pages 145–152 Manchester, August 2008 Mind the Gap: Dangers of Divorcing Evaluations of Summary Content from Linguistic Quality John M. Conroy IDA/Center for Computing Sciences Bowie, Maryland, USA conroy@super.org Hoa Trang Dang Information Access Division National Institute of Standards and Technology Gaithersburg, Maryland, USA hoa.dang@nist.gov Abstract In this paper, we analyze the state of cur- rent human and automatic evaluation of topic-focused summarization in the Docu- ment Understanding Conference main task for 2005-2007. The analyses show that while ROUGE has very strong correlation with responsiveness for both human and automatic summaries, there is a signifi- cant gap in responsiveness between hu- mans and systems which is not accounted for by the ROUGE metrics. In addition to teasing out gaps in the current auto- matic evaluation, we propose a method to maximize the strength of current auto- matic evaluations by using the method of canonical correlation. We apply this new evaluation method, which we call ROSE (ROUGE Optimal Summarization Evalua- tion), to find the optimal linear combina- tion of ROUGE scores to maximize corre- lation with human responsiveness. 1 Introduction ROUGE (Lin, 2004) and its linguistically- motivated descendent, Basic Elements (BE) (Hovy et al., 2005), evaluate a summary by computing its overlap with a set of model (human) summaries; ROUGE considers lexical n-grams as the unit for comparing the overlap between summaries, while Basic Elements uses larger units of com- parison based on the output of syntactic parsers. The ROUGE/BE toolkit has become the standard automatic method for evaluating the content of c⃝ 2008. Licensed under the Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported li- cense (http://creativecommons.org/licenses/by-nc-sa/3.0/). Some rights reserved. machine-generated summaries, but the correlation of these automatic scores with human evaluation metrics has not always been consistent. In this paper, we analyze the state of current human and automatic evaluation of topic-focused summarization. Using the results of the Document Understanding Conference main task for 2005- 2007 we explore the correlation between variants of ROUGE and the human metrics of responsive- ness and linguistic quality. The analyses expose a number of challenges and several surprising re- sults. In particular, while ROUGE has very strong correlation with responsiveness for both human and system summaries, there is a significant gap in responsiveness between humans and systems which is not accounted for by the ROUGE metrics. One cause of the gap is that many automatic sum- marizers truncate the last sentence of their sum- mary, which shows significant reduction in the re- sponsiveness score but does not result in a statis- tically significant drop in ROUGE scores. In ad- dition to teasing out gaps in the current automatic evaluation, we propose a method to maximize the strength of current automatic evaluations by us- ing the method of canonical correlation. We apply this new evaluation method, which we call ROSE (ROUGE Optimal Summarization Evaluation), to find the optimal linear combination of ROUGE metrics to maximize correlation with human re- sponsiveness. 2 DUC 2005-2007 Task and …

Content truncated to 500 words