The Feasibility of Embedding Based Automatic Evaluation for Single Document Summarization

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 1216–1221, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 1216 The Feasibility of Embedding Based Automatic Evaluation for Single Document Summarization Simeng Sun University of Pennsylvania simsun@seas.upenn.edu Ani Nenkova University of Pennsylvania nenkova@seas.upenn.edu Abstract ROUGE is widely used to automatically evaluate summarization systems. However, ROUGE measures semantic overlap between a system summary and a human reference on word-string level, much at odds with the contemporary treatment of semantic meaning. Here we present a suite of experiments on us- ing distributed representations for evaluating summarizers, both in reference-based and in reference-free setting. Our experimental re- sults show that the max value over each dimen- sion of the summary ELMo word embeddings is a good representation that results in high correlation with human ratings. Averaging the cosine similarity of all encoders we tested yields high correlation with manual scores in reference-free setting. The distributed repre- sentations outperform ROUGE in recent cor- pora for abstractive news summarization but are less good on older test data and systems. 1 Introduction The widely used ROUGE (Lin, 2004) automatic evaluation for summarization relies on token over- lap between reference and system summary. This limited view of meaning has motivated numer- ous studies on summarization evaluation (Zhou et al., 2006; Ganesan, 2018; ShafieiBavani et al., 2018), and the related areas of translation and dia- log, to explore more compelling semantic match- ing (Kauchak and Barzilay, 2006; Lavie and Denkowski, 2009; Lo and Wu, 2011; Chen and Guo, 2015; Liu et al., 2016; Tao et al., 2018). Most recently, incorporating word embeddings in ROUGE pairwise comparison of n-grams has proven beneficial (Ng and Abrecht, 2015), as well as representing sentences using universal sentence representation to predict the quality of translation (Shimanaka et al., 2018). We build upon this line of work and show that cosine similarity between the reference and summary embedding works well, and better than ROUGE on recent datasets, for comparing sin- gle document summarization systems. Unlike prior work (Ng and Abrecht, 2015), we thoroughly abandon ROUGE and n-gram co-occurrences in the computation of semantic similarity. To give a sense of the generalizability of our findings, we validate the method on three different test sets with human evaluation. We compare several pop- ular representation including sentence embedding, un-contextualized word embedding and contex- tualized word embedding. Finally, we present experiments on evaluating single document sum- maries without reference summaries which was originally proposed for multi-document summa- rization (Louis and Nenkova, 2013) and explored a variety of word-string similarity techniques. Here we study reference-free evaluations via embed- ding similarity between the full document to be summarized and the system summaries. 2 Embeddings To get a dense low-dimenional representation of texts, we test seven representations covering sen- tence embedding, variants of un-contextualized word embedding and variants of contextualized word embedding. Specifically: (i) Two Google universal sentence encoders: (Cer et al., 2018), an encoder …

Content truncated to 500 words