Evaluation of Review Summaries via Question-Answering

Evaluation of Review Summaries via Question-Answering Nannan Huang RMIT University, Australia s3754491@student.rmit.edu.au Xiuzhen Zhang RMIT University, Australia xiuzhen.zhang@rmit.edu.au Abstract Summarisation of reviews aims at compress- ing opinions expressed in multiple review doc- uments into a concise form while still cover- ing the key opinions. Despite the advancement in summarisation models, evaluation metrics for opinionated text summaries lag behind and still rely on lexical-matching metrics such as ROUGE. In this paper, we propose using the question-answering(QA) approach to evaluate summaries of opinions in reviews. We pro- pose to identify opinion-bearing text spans in the reference summary to generate QA pairs so as to capture salient opinions. A QA model is then employed to probe the candidate summary to evaluate information overlap be- tween the candidate and reference summaries. We show that our metric RunQA, Review Summary Evaluation via Question Answering, correlates well with human judgments in terms of coverage and focus of information. 1 Introduction Opinion summarisation takes input documents like online reviews or social media posts where users express their opinions on topics and condense them into a single piece of text. The summary should reflect the core opinions expressed in the source documents. Recent studies in opinion summari- sation have shown an advancement moving from extractive (Meng et al., 2012; Ku et al., 2006; i.a), i.e. copying sections from the original reviews to produce a summary, to abstractive (Chu and Liu, 2019; Braˇzinskas et al., 2020a; Braˇzinskas et al., 2020b; i.a), i.e. generating new phrases that reflect the information covered in the original text. Despite the advancement in summarisation mod- els, evaluation metrics for opinionated text sum- maries lag behind. For evaluation of review sum- maries, traditional token-matching ROUGE (Lin, 2004) is still widely used, supplemented with hu- man evaluation on the relative ranking of system- generated summaries in terms of quality dimen- sions such as Fluency, Coherence, Non-redundancy, Informativeness, and Sentiment (Chu and Liu, 2019; Braˇzinskas et al., 2020a; Braˇzinskas et al., 2020b). It is well understood that ROUGE cannot capture the same meaning expressed in different token sequences. Neural model-based metrics have been proposed for general text summarisation evaluation. Espe- cially QAEval (Deutsch et al., 2021) has been pro- posed for evaluating the information quality of ab- stractive summaries with respect to the reference summaries. A key step for the success of QAEval for summarisation evaluation is extracting answers and generating questions covering a significant amount of important Summarisation Content Units (SCUs). Generally, noun phrases(NP) and named entities(NER) are used to generate question-answer pairs in QA models (Durmus et al., 2020; Wang et al., 2020; Deutsch et al., 2021), but their appli- cability for review summary evaluation is yet to be examined. In this paper, we propose to evaluate review sum- maries with question answering (QA) based on neural models, with a focus on evaluating the in- formation quality of review summaries. Modern abstractive summarisation systems can generate sentences of high linguistic quality – grammati- cally correct, easy to read and understand– but it is more important …

Content truncated to 500 words