Can You Summarize This? Identifying Correlates of Input Difficulty for Multi-Document Summarization

Proceedings of ACL-08: HLT, pages 825–833, Columbus, Ohio, USA, June 2008. c⃝2008 Association for Computational Linguistics Can you summarize this? Identifying correlates of input difficulty for generic multi-document summarization Ani Nenkova University of Pennsylvania Philadelphia, PA 19104, USA nenkova@seas.upenn.edu Annie Louis University of Pennsylvania Philadelphia, PA 19104, USA lannie@seas.upenn.edu Abstract Different summarization requirements could make the writing of a good summary more dif- ficult, or easier. Summary length and the char- acteristics of the input are such constraints in- fluencing the quality of a potential summary. In this paper we report the results of a quanti- tative analysis on data from large-scale evalu- ations of multi-document summarization, em- pirically confirming this hypothesis. We fur- ther show that features measuring the cohe- siveness of the input are highly correlated with eventual summary quality and that it is possi- ble to use these as features to predict the diffi- culty of new, unseen, summarization inputs. 1 Introduction In certain situations even the best automatic sum- marizers or professional writers can find it hard to write a good summary of a set of articles. If there is no clear topic shared across the input articles, or if they follow the development of the same event in time for a longer period, it could become difficult to decide what information is most representative and should be conveyed in a summary. Similarly, length requirements could pre-determine summary quality—a short outline of a story might be confus- ing and unclear but a page long discussion might give an excellent overview of the same issue. Even systems that perform well on average pro- duce summaries of poor quality for some inputs. For this reason, understanding what aspects of the in- put make it difficult for summarization becomes an interesting and important issue that has not been ad- dressed in the summarization community untill now. In information retrieval, for example, the variable system performance has been recognized as a re- search challenge and numerous studies on identify- ing query difficulty have been carried out (most re- cently (Cronen-Townsend et al., 2002; Yom-Tov et al., 2005; Carmel et al., 2006)). In this paper we present results supporting the hy- potheses that input topicality cohesiveness and sum- mary length are among the factors that determine summary quality regardless of the choice of summa- rization strategy (Section 2). The data used for the analyses comes from the annual Document Under- standing Conference (DUC) in which various sum- marization approaches are evaluated on common data, with new test sets provided each year. In later sections we define a suite of features cap- turing aspects of the topicality cohesiveness of the input (Section 3) and relate these to system perfor- mance, identifying reliable correlates of input diffi- culty (Section 4). Finally, in Section 5, we demon- strate that the features can be used to build a clas- sifier predicting summarization input difficulty with accuracy considerably above chance level. 2 Preliminary analysis and distinctions: DUC 2001 Generic multi-document summarization was fea- tured as a …

Content truncated to 500 words