Automated Pyramid Summarization Evaluation

Proceedings of the 23rd Conference on Computational Natural Language Learning, pages 404–418 Hong Kong, China, November 3-4, 2019. c⃝2019 Association for Computational Linguistics 404 Automated Pyramid Summarization Evaluation Yanjun Gao1, Chen Sun2, and Rebecca J. Passonneau1 1,2Department of Computer Science and Engineering Pennsylvania State University 1 {yug125,rjp49}@cse.psu.edu 2chensunx@gmail.com Abstract Pyramid evaluation was developed to assess the content of paragraph length summaries of source texts. A pyramid lists the distinct units of content found in several reference sum- maries, weights content units by how many reference summaries they occur in, and pro- duces three scores based on the weighted con- tent of new summaries. We present an au- tomated method that is more efficient, more transparent, and more complete than previous automated pyramid methods. It is tested on a new dataset of student summaries, and histori- cal NIST data from extractive summarizers. 1 Introduction During the 70’s and 80’s, educational pychologists studied human summarization skills, and their de- velopment throughout secondary school and be- yond. Three separate skills are acquired in the following order: selection of important informa- tion, abstraction through vocabulary generaliza- tion and sentence fusion, and integration with background knowledge (van Dijk and Kintsch, 1977; Brown and Day, 1983). A recent com- parison of summaries from human experts ver- sus extractive summarizers on forty-six topics from the TAC 2010 summarization challenge used automatic caseframe analysis, and found essen- tially these same properties in the human sum- maries, and not in the extractive ones (Cheung and Penn, 2013). Abstractive summarizers, how- ever, are beginning to replicate the first two of these behaviors, as illustrated in many published examples based on encoder-decoder and pointer- generator neural architectures (Nallapati et al., 2016; See et al., 2017; Hsu et al., 2018; Guo et al., 2018). Summarization evaluation relies almost ex- clusively on ROUGE (Lin, 2004), an automated tool that cannot directly assess importance of sum- mary content, or novel wording for the same infor- Aligned PyrEval (W=5) and Manual (W=4) SCU RSUM1 For example, an art gallery in London held an exhibit. with digital curr. as the preferred . . . RSUM2 However, there has been some positive news as bus. such as a Scottish Hotel & a London Art Gallery are allowing cust. to pay with crypto currencies RSUM3 Cellan-Jones (2018) writes recent days both a London art gallery and a Scottish hotel . . . to allow their cust. to pay with crypto-currencies. RSUM4 by suggesting that {a London art gallery & Scottish hotel chain plan to . . . support for different crypto-currencies.}P araph . . . { that the London based art gallery would use only crypto currencies}P araph RSUM5 Businesses located in London and Scotland have made enquiries to allow payment from customers using cryptoc. Match to a student summary that used synomyms: a craftsmanship exhibition alongside a Scottish inn have plans for their clients to pay in digital currencies Figure 1: Alignment of a single PyrEval SCU of weight 5 to a manual SCU of weight 4 …

Content truncated to 500 words