PyrEval: An Automated Method for Summary Content Analysis

PyrEval: An Automated Method for Summary Content Analysis Yanjun Gao, Andrew Warner and Rebecca J. Passonneau The Pennsylvania State University Pennsylvania, USA yug125@cse.psu.edu, acw5456@psu.edu, rjp49@cse.psu.edu Abstract The pyramid method is a content analysis approach in automatic summarization evaluation for manual construction of a content model from reference summaries, and manual scoring of unseen summaries with the pyramid model. PyrEval automates the manual pyramid method. PyrEval uses low-dimension distributional semantics to represent phrase meanings, and a new algorithm, EDUA (Emergent Discovery of Units of Attraction), to solve a set cover problem to construct the content model from vectorized phrases. Because the vectors are pretrained, and EDUA is an efficient greedy algorithm, PyrEval can apply pyramid content evaluation with no retraining, and in excellent time. Moreover, PyrEval has been tested on many datasets derived from humans and machine generated summaries, and shown good performance on both. Keywords: Content Evaluation, Summarization, Summarization Evaluation Tool 1. Introduction Automatic summarization methods that produce a para- graph to express the main ideas of one or more texts have shifted in recent years. The long-standing prevalence of extractive summarizers, which select complete sentences from source documents, has begun to give way to abstrac- tive summarizers, which rewrite or generate sentences to eliminate less important content (Bing et al., 2015; Rush et al., 2015; Liu et al., 2015; Durrett et al., 2016; Nema et al., 2017). Automatic content evaluation of summarization systems has not seen the same progress. ROUGE has long been the dominant method for evaluating summary con- tent, and is used in most of the papers cited above. Here we present PyrEval, which applies pyramid content anal- ysis (Nenkova and Passonneau, 2004), a method intended for evaluation of abstractive summarization. The main in- novation in PyrEval is a novel formulation of the pyramid construction problem, and a greedy algorithm whose ap- proximate solution is both efficient and effective. The pyramid method, introduced in (Nenkova and Passon- neau, 2004), groups the distinct ideas (content units) men- tioned in reference summaries to create a pyramid model of content, and differentiates the content units by impor- tance as well as meaning. It then evaluates target sum- maries against the pyramid content model. Early work ap- plied manual annotation of the pyramid method for largely extractive automated summarizers at a series of Document Understanding Conferences (DUC) and Text Analysis Con- ferences (TAC) administered by the National Institute of Standards (NIST) from 2006 through 2011 (Passonneau et al., 2006; Varma et al., 2009; Ji et al., 2010). Figure 1 shows a content unit from one of the pyramids created for the NIST pyramid evaluation at DUC 2006, along with a sentence in a summary from a peer (automated summarizer) that was matched to the content unit (CU). The illustrated content unit, labeled by the annotator, lists a phrase from each of three reference summaries (D, E, H) that express the labeled meaning. The importance weight of the CU is the number of contributing reference summaries. Automated pyramid methods such as PyrScore …

Content truncated to 500 words