Automatic Evaluation of Linguistic Quality in Multi-Document Summarization

Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 544–554, Uppsala, Sweden, 11-16 July 2010. c⃝2010 Association for Computational Linguistics Automatic Evaluation of Linguistic Quality in Multi-Document Summarization Emily Pitler, Annie Louis, Ani Nenkova Computer and Information Science University of Pennsylvania Philadelphia, PA 19104, USA epitler,lannie,nenkova@seas.upenn.edu Abstract To date, few attempts have been made to develop and validate methods for au- tomatic evaluation of linguistic quality in text summarization. We present the first systematic assessment of several diverse classes of metrics designed to capture var- ious aspects of well-written text. We train and test linguistic quality models on con- secutive years of NIST evaluation data in order to show the generality of results. For grammaticality, the best results come from a set of syntactic features. Focus, coher- ence and referential clarity are best evalu- ated by a class of features measuring local coherence on the basis of cosine similarity between sentences, coreference informa- tion, and summarization specific features. Our best results are 90% accuracy for pair- wise comparisons of competing systems over a test set of several inputs and 70% for ranking summaries of a specific input. 1 Introduction Efforts for the development of automatic text sum- marizers have focused almost exclusively on im- proving content selection capabilities of systems, ignoring the linguistic quality of the system out- put. Part of the reason for this imbalance is the existence of ROUGE (Lin and Hovy, 2003; Lin, 2004), the system for automatic evaluation of con- tent selection, which allows for frequent evalua- tion during system development and for report- ing results of experiments performed outside of the annual NIST-led evaluations, the Document Understanding Conference (DUC)1 and the Text Analysis Conference (TAC)2. Few metrics, how- ever, have been proposed for evaluating linguistic 1http://duc.nist.gov/ 2http://www.nist.gov/tac/ quality and none have been validated on data from NIST evaluations. In their pioneering work on automatic evalua- tion of summary coherence, Lapata and Barzilay (2005) provide a correlation analysis between hu- man coherence assessments and (1) semantic re- latedness between adjacent sentences and (2) mea- sures that characterize how mentions of the same entity in different syntactic positions are spread across adjacent sentences. Several of their models exhibit a statistically significant agreement with human ratings and complement each other, yield- ing an even higher correlation when combined. Lapata and Barzilay (2005) and Barzilay and Lapata (2008) both show the effectiveness of entity-based coherence in evaluating summaries. However, fewer than five automatic summarizers were used in these studies. Further, both sets of experiments perform evaluations of mixed sets of human-produced and machine-produced sum- maries, so the results may be influenced by the ease of discriminating between a human and ma- chine written summary. Therefore, we believe it is an open question how well these features predict the quality of automatically generated summaries. In this work, we focus on linguistic quality eval- uation for automatic systems only. We analyze how well different types of features can rank good and poor machine-produced summaries. Good performance on …

Content truncated to 500 words