Do You Hear The People Sing? Key Point Analysis via Iterative Clustering and Abstractive Summarisation

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 14064–14080 July 9-14, 2023 ©2023 Association for Computational Linguistics Do You Hear The People Sing? Key Point Analysis via Iterative Clustering and Abstractive Summarisation Hao Li♣ Viktor Schlegel♢♣ Riza Batista-Navarro♣ and Goran Nenadic♣ ♣Department of Computer Science, University of Manchester, United Kingdom ♢ASUS Intelligent Cloud Services (AICS), Singapore {hao.li-2, riza.batista, gnenadic}@manchester.ac.uk viktor_schlegel@asus.com Abstract Argument summarisation is a promising but currently under-explored field. Recent work has aimed to provide textual summaries in the form of concise and salient short texts, i.e., key points (KPs), in a task known as Key Point Analysis (KPA). One of the main challenges in KPA is finding high-quality key point candi- dates from dozens of arguments even in a small corpus. Furthermore, evaluating key points is crucial in ensuring that the automatically gen- erated summaries are useful. Although auto- matic methods for evaluating summarisation have considerably advanced over the years, they mainly focus on sentence-level compar- ison, making it difficult to measure the quality of a summary (a set of KPs) as a whole. Ag- gravating this problem is the fact that human evaluation is costly and unreproducible. To ad- dress the above issues, we propose a two-step abstractive summarisation framework based on neural topic modelling with an iterative cluster- ing procedure, to generate key points which are aligned with how humans identify key points. Our experiments show that our framework ad- vances the state of the art in KPA, with per- formance improvement of up to 14 (absolute) percentage points, in terms of both ROUGE and our own proposed evaluation metrics1. Fur- thermore, we evaluate the generated summaries using a novel set-based evaluation toolkit. Our quantitative analysis demonstrates the effective- ness of our proposed evaluation metrics in as- sessing the quality of generated KPs. Human evaluation further demonstrates the advantages of our approach and validates that our proposed evaluation metric is more consistent with hu- man judgment than ROUGE scores. 1 Introduction Automated summarisation of salient arguments from texts is a long-standing problem, which has 1Our code can be found on Github: https://github. com/HarrywillDr/keypoint-Analysis attracted a lot of research interest in the last decade. Early efforts proposed to tackle argument summari- sation as a clustering task, implicitly expressing the main idea based on different notions of relatedness, such as argument facets (Misra et al., 2016), sim- ilarity (Reimers et al., 2019) and frames (Ajjour et al., 2019). However, they do not create easy-to- understand summaries from clusters, which leads to unmitigated challenges in comprehensively nav- igating the overwhelming wealth of information available in online textual content. Recent trends aim to alleviate this problem by summarising a large collection of arguments in the form of a set of concise sentences that describe the collection at a high-level—these sentences are called key points (KPs). This approach was first proposed by Bar-Haim et al. (2020a), consisting of two subtasks, namely, key point generation (se- lecting key point arguments from the corpus) …

Content truncated to 500 words