Estimating Summary Quality with Pairwise Preferences

Proceedings of NAACL-HLT 2018, pages 1687–1696 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Estimating Summary Quality with Pairwise Preferences Markus Zopf Research Training Group AIPHES / Knowledge Engineering Group Department of Computer Science, Technische Universit¨at Darmstadt Hochschulstraße 10, 64289 Darmstadt, Germany zopf@aiphes.tu-darmstadt.de Abstract Automatic evaluation systems in the field of automatic summarization have been relying on the availability of gold standard summaries for over ten years. Gold standard summaries are expensive to obtain and often require the availability of domain experts to achieve high quality. In this paper, we propose an alter- native evaluation approach based on pairwise preferences of sentences. In comparison to gold standard summaries, they are simpler and cheaper to obtain. In our experiments, we show that humans are able to provide useful feedback in the form of pairwise preferences. The new framework performs better than the three most popular versions of ROUGE with less expensive human input. We also show that our framework can reuse already available evaluation data and achieve even better results. 1 Introduction Due to the huge amount of information contained in texts, the task of automatic text summarization (Mani, 2001; Nenkova and McKeown, 2011) is a pressing challenge nowadays and will become even more important in the future. Building sum- marization systems is, however, not the only chal- lenge in this field. Evaluation of automatically generated summaries is also an active field of re- search. Ideally, we would like to ask humans for their opinion about the quality of automatically gener- ated summaries in an extrinsic evaluation (Hal- teren and Teufel, 2003). Since summaries are gen- erated for humans, they should also be evaluated directly by humans. Unfortunately, manual evalu- ation cannot be performed at a large scale because of the huge effort which is necessary for evalua- tion. (Lin, 2004) reported that 3,000 hours of hu- man effort would be required for a simple evalu- ation of the summaries for the Document Under- standing Conference (DUC) 2003, a popular sum- marization shared task series. This motivates re- search of automatic evaluation methods for auto- matic summarization. ROUGE (Lin, 2004), the current method of choice for evaluating automated text summariza- tion, relies on the availability of gold standard summaries. The gold standard summaries are used to define the optimal output of a summarization system. Writing high-quality summaries, how- ever, requires the availability of expert writers and takes a lot of effort. (Dang, 2005) reported that creating the reference summaries for the DUC 2005 shared task was a difficult endeavor with an effort of five hours to produce each reference sum- mary. Since ROUGE needs at least four reference summaries to become reasonably reliable, the ef- fort sums up to at least 20 hours of annotation ef- fort per topic. For this reason, gold standard sum- maries are only available for a few, rather small datasets. Also the more accurate (but also even more expensive) Pyramid method (Nenkova and Passonneau, 2004) requires expensive gold stan- dard summaries. …

Content truncated to 500 words