Red-faced ROUGE: Examining the Suitability of ROUGE for Opinion Summary Evaluation

Red-faced ROUGE: Examining the Suitability of ROUGE for Opinion Summary Evaluation Wenyi Tay1,2, Aditya Joshi2, Xiuzhen Zhang1, Sarvnaz Karimi2 and Stephen Wan2 1RMIT University, Australia 2CSIRO Data61, Australia {wenyi.tay, xiuzhen.zhang}@rmit.edu.au {Aditya.Joshi, Sarvnaz.Karimi, Stephen.Wan}@data61.csiro.au Abstract One of the most common metrics to automat- ically evaluate opinion summaries is ROUGE, a metric developed for text summarisation. ROUGE counts the overlap of word or word units between candidate summaries and refer- ence summaries. This formulation treats all words in the reference summary equally. In opinion summaries, however, not all words in the reference are equally important. Opin- ion summarisation requires to correctly pair two types of semantic information: (1) opin- ion target, or aspect; and, (2) polarity of can- didate and reference summaries. We inves- tigate the suitability of ROUGE for evaluat- ing opinion summaries of online reviews. We design three experiments to evaluate the be- haviour of ROUGE for opinion summarisa- tion on the ability to capture aspect and po- larity. We show that ROUGE cannot distin- guish opinion summaries of the same or op- posite polarities for the same aspect. More- over, ROUGE scores have significant variance under different configuration settings. As a re- sult, we present three recommendations for fu- ture work on evaluating opinion summaries. 1 Introduction Popular e-commerce websites allow users to ex- press their opinion about products or services in the form of reviews. An opinion is formally defined as a combination of aspect (an attribute of the product or service as the opinion target, expressed through aspect words), and sentiment polarity (either positive or negative, expressed through opinion words) (Liu, 2012). The opin- ion expressed in online reviews potentially helps prospective buyers to make decisions. Given the large volume of reviews, it is time-consuming and often impractical for a user of these websites to read all reviews pertaining to the set of products that they are considering to purchase. This makes opinion summarisation important because it al- lows users to obtain aggregate key opinions about the product or service based on its reviews. Given the value of opinion summaries, automatic opinion summarisation is an active area of research with the focus of producing high-quality opinion sum- maries. The quality of an opinion summary would ide- ally be evaluated by human annotators. For ex- ample, annotators may read a summary and rate it according to quality measures such as informa- tiveness, ability to capture sentiment polarity, co- herence and redundancy (Angelidis and Lapata, 2018). However, human evaluation is resource- intensive and not scalable. This motivates auto- matic evaluation. In the case of automatic evalua- tion, for a set of product reviews, reference sum- maries are written by human experts apriori. Ref- erence summaries are the ground truth summaries against which candidate summaries to be evalu- ated. Using reference and candidate summaries, au- tomatic evaluation of opinion summaries adopts metrics from text summarisation (Lin, 2004) and machine translation (Papineni et al., 2002; Lavie and Denkowski, 2009). We focus on the most frequently reported metric for opinion summari- sation, …

Content truncated to 500 words