Crowdsourcing Lightweight Pyramids for Manual Summary Evaluation

Proceedings of NAACL-HLT 2019, pages 682–687 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 682 Crowdsourcing Lightweight Pyramids for Manual Summary Evaluation Ori Shapira1, David Gabay1, Yang Gao2, Hadar Ronen1, Ramakanth Pasunuru3, Mohit Bansal3, Yael Amsterdamer1, and Ido Dagan1 1Bar-Ilan University 2UKP Technische Universit¨at Darmstadt {obspp18, dawid.gabay}@gmail.com3UNC Chapel Hill gao@ukp.informatik.tu-darmstadt.de, hadarg@gmail.com {ram, mbansal}@cs.unc.edu, {amstery, dagan}@cs.biu.ac.il Abstract Conducting a manual evaluation is consid- ered an essential part of summary evalua- tion methodology. Traditionally, the Pyramid protocol, which exhaustively compares sys- tem summaries to references, has been per- ceived as very reliable, providing objective scores. Yet, due to the high cost of the Pyramid method and the required expertise, researchers resorted to cheaper and less thorough man- ual evaluation methods, such as Responsive- ness and pairwise comparison, attainable via crowdsourcing. We revisit the Pyramid ap- proach, proposing a lightweight sampling- based version that is crowdsourcable. We an- alyze the performance of our method in com- parison to original expert-based Pyramid eval- uations, showing higher correlation relative to the common Responsiveness method. We release our crowdsourced Summary-Content- Units, along with all crowdsourcing scripts, for future evaluations. 1 Introduction Evaluating content quality of summaries is an in- tegral part of summarization research. Measur- ing the performance of a summarization system can be done through either automatic or manual evaluation. An automatic evaluation, in practice working at the lexical level, provides an inexpen- sive means of measuring the validity of a sys- tem, both for system comparisons and for quick development cycle testing. Due to the shallow- ness of the automatic approaches, their reliabil- ity is often perceived as insufficient (Owczarzak et al., 2012; Chaganty et al., 2018). This calls for the more expensive manual evaluation, which em- ploys human-in-the-loop protocols for assessment. The Pyramid method (Nenkova and Passon- neau, 2004) is a prominent manual evaluation methodology that is considered highly reliable for comparing summarization systems. It relies on a small set of manually-crafted reference sum- maries, out of which all summary content units (SCUs) are manually extracted. System sum- maries are then manually checked for coverage of each individual SCU, from which an overall system score is derived. The Pyramid evaluation method’s reliability comes at a cost. It requires laborious manual work performed by annotators who must browse through non-trivial guidelines (Passonneau, 2006). Due to these drawbacks, it was only used in a few DUC and TAC (NIST, 2014, 2018) benchmarks. Instead, summarization work in recent years has mostly employed simpler manual evaluation ap- proaches, such as Responsiveness and pairwise comparison, which do not rely on reference sum- maries and can be attained via crowdsourcing. Yet, these methods are quite subjective, since eval- uators need to provide only a single global judg- ment for the quality of a summary (or a pair of summaries). Such judgments are far more subjec- tive than the Pyramid score, which is derived from many, more objective, local decisions, each judg- ing independently the presence of an individual SCU. Indeed, it was shown that …

Content truncated to 500 words