Better Summarization Evaluation with Word Embeddings for ROUGE

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1925–1930, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Better Summarization Evaluation with Word Embeddings for ROUGE Jun-Ping Ng Bloomberg L.P. New York, USA jng324@bloomberg.net Viktoria Abrecht Bloomberg L.P. New York, USA vkanchakousk@bloomberg.net Abstract ROUGE is a widely adopted, automatic evaluation measure for text summariza- tion. While it has been shown to corre- late well with human judgements, it is bi- ased towards surface lexical similarities. This makes it unsuitable for the evalua- tion of abstractive summarization, or sum- maries with substantial paraphrasing. We study the effectiveness of word embed- dings to overcome this disadvantage of ROUGE. Specifically, instead of measur- ing lexical overlaps, word embeddings are used to compute the semantic similarity of the words used in summaries instead. Our experimental results show that our pro- posal is able to achieve better correlations with human judgements when measured with the Spearman and Kendall rank co- efficients. 1 Introduction Automatic text summarization is a rich field of re- search. For example, shared task evaluation work- shops for summarization were held for more than a decade in the Document Understanding Con- ference (DUC), and subsequently the Text Anal- ysis Conference (TAC). An important element of these shared tasks is the evaluation of participating systems. Initially, manual evaluation was carried out, where human judges were tasked to assess the quality of automatically generated summaries. However in an effort to make evaluation more scaleable, the automatic ROUGE1 measure (Lin, 2004b) was introduced in DUC-2004. ROUGE determines the quality of an automatic summary through comparing overlapping units such as n- grams, word sequences, and word pairs with hu- man written summaries. 1Recall-Oriented Understudy of Gisting Evaluation ROUGE is not perfect however. Two problems with ROUGE are that 1) it favors lexical simi- larities between generated summaries and model summaries, which makes it unsuitable to evaluate abstractive summarization, or summaries with a significant amount of paraphrasing, and 2) it does not make any provision to cater for the readability or fluency of the generated summaries. There has been on-going efforts to improve on automatic summarization evaluation measures, such as the Automatically Evaluating Summaries of Peers (AESOP) task in TAC (Dang and Owczarzak, 2009; Owczarzak, 2010; Owczarzak and Dang, 2011). However, ROUGE remains as one of the most popular metric of choice, as it has repeatedly been shown to correlate very well with human judgements (Lin, 2004a; Over and Yen, 2004; Owczarzak and Dang, 2011). In this work, we describe our efforts to tackle the first problem of ROUGE that we have iden- tified above — its bias towards lexical similari- ties. We propose to do this by making use of word embeddings (Bengio et al., 2003). Word embed- dings refer to the mapping of words into a multi- dimensional vector space. We can construct the mapping, such that the distance between two word projections in the vector space corresponds to the semantic similarity between the two words. By in- corporating …

Content truncated to 500 words