Ranking Human and Machine Summarization Systems

Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 467–473, Edinburgh, Scotland, UK, July 27–31, 2011. c⃝2011 Association for Computational Linguistics Ranking Human and Machine Summarization Systems Peter Rankel University of Maryland College Park, Maryland rankel@math.umd.edu John M. Conroy IDA/Center for Computing Sciences Bowie, Maryland conroyjohnm@gmail.com Eric V. Slud University of Maryland College Park, Maryland evs@math.umd.edu Dianne P. O’Leary University of Maryland College Park, Maryland oleary@cs.umd.edu Abstract The Text Analysis Conference (TAC) ranks summarization systems by their average score over a collection of document sets. We in- vestigate the statistical appropriateness of this score and propose an alternative that better distinguishes between human and machine evaluation systems. 1 Introduction For the past several years, the National Institute of Standards and Technology (NIST) has hosted the Text Analysis Conference (TAC) (previously called the Document Understanding Conference (DUC)) (Nat, 2010). A major theme of this conference is multi-document summarization: machine summa- rization of sets of related documents, sometimes query-focused and sometimes generic. The sum- marizers are judged by how well the summaries match human-generated summaries in either auto- matic metrics such as ROUGE (Lin and Hovy, 2003) or manual metrics such as responsiveness or pyra- mid evaluation (Nenkova et al., 2007). Typically the systems are ranked by their average score over all document sets. Ranking by average score is quite appropriate un- der certain statistical hypotheses, for example, when each sample is drawn from a distribution which differs from the distribution of other samples only through a location shift (Randles and Wolfe, 1979). However, a non-parametric (rank-based) analysis of variance on the summarizers’ scores on each docu- ment set revealed an impossibly small p-value (less Figure 1: Confidence Intervals from a non-parametric Tukey’s honestly significant difference test for 46 TAC 2010 update document sets. The blue confidence interval (for document set d1032) does not overlap any of the 30 red intervals. Hence, the test concludes that 30 document sets have mean significantly different from the mean of d1032. 467 Figure 2: Overall Responsiveness scores. Figure 3: Linguistic scores. Figure 4: Pyramid scores. Figure 5: ROUGE-2 scores for the TAC 2010 update summary task, organized by document set (y-axis) and summarizer (x-axis). The 51 summarizers fall into two distinct groups: machine systems (first 43 columns) and humans (last 8 columns). Note that each human only summarized half of the document sets, thus creating 23 missing values in each of the last 8 columns. Black is used to indicate missing values in the last 8 columns and low scores in the first 43 columns. than 10−12 using Matlab’s kruskalwallis 1), providing evidence that a summary’s score is not independent of the document set. This effect can be seen in Figure 1, showing the confidence bands, as computed by a Tukey honestly significant differ- ence test for each document set’s difficulty as mea- sured by the mean rank responsiveness score for TAC 2010. The test clearly shows that the summa- rizer performances on different document sets have different averages. We further …

Content truncated to 500 words