Revisiting Summarization Evaluation for Scientific Articles

Revisiting Summarization Evaluation for Scientific Articles Arman Cohan and Nazli Goharian Information Retrieval Lab Department of Computer Science Georgetown University arman@ir.cs.georgetown.edu, nazli@ir.cs.georgetown.edu Abstract Evaluation of text summarization approaches have been mostly based on metrics that measure similarities of system generated summaries with a set of human written gold-standard summaries. The most widely used metric in summarization evaluation has been the ROUGE family. ROUGE solely relies on lexical overlaps between the terms and phrases in the sentences; therefore, in cases of terminology variations and paraphrasing, ROUGE is not as effective. Scientific article summarization is one such case that is different from general domain summarization (e.g. newswire data). We provide an extensive analysis of ROUGE’s effectiveness as an evaluation metric for scientific summarization; we show that, contrary to the common belief, ROUGE is not much reliable in evaluating scientific summaries. We furthermore show how different variants of ROUGE result in very different correlations with the manual Pyramid scores. Finally, we propose an alternative metric for summarization evaluation which is based on the content relevance between a system generated summary and the corresponding human written summaries. We call our metric SERA (Summarization Evaluation by Relevance Analysis). Unlike ROUGE, SERA consistently achieves high correlations with manual scores which shows its effectiveness in evaluation of scientific article summarization. Keywords: Summarization, Evaluation, Scientific articles 1. Introduction Automatic text summarization has been an active research area in natural language processing for several decades. To compare and evaluate the performance of different summarization systems, the most intuitive approach is assessing the quality of the summaries by human evaluators. However, manual evaluation is expensive and the obtained results are subjective and difficult to reproduce (Giannakopoulos and Karkaletsis, 2013). To address these problems, automatic evaluation measures for summarization have been proposed. ROUGE (Lin, 2004) is one of the first and most widely used metrics in summarization evaluation. It facilitates evaluation of system generated summaries by comparing them to a set of human written gold-standard summaries. It is inspired by the success of a similar metric BLEU (Papineni et al., 2002) which is being used in Machine Translation (MT) evaluation. The main success of ROUGE is due to its high correlation with human assessment scores on standard benchmarks (Lin, 2004). ROUGE has been used as one of the main evaluation metrics in later summarization benchmarks such as TAC1 (Owczarzak and Dang, 2011). Since the establishment of ROUGE, almost all research in text summarization have used this metric as the main means for evaluating the quality of the proposed approaches. The public availability of ROUGE as a toolkit for summarization evaluation has contributed to its wide usage. While ROUGE has originally shown good correlations with human assessments, the study of its effectiveness was only limited to a few benchmarks on news summarization data (DUC2 2001-2003 benchmarks). Since 2003, summarization 1Text Analysis Conference (TAC) is a series of workshops for evaluating research in Natural Language Processing has grown to much further domains and genres such as scientific documents, social media and question answering. …

Content truncated to 500 words