Semantic Overlap Summarization among Multiple Alternative Narratives: An Exploratory Study

Proceedings of the 29th International Conference on Computational Linguistics, pages 6195–6207 October 12–17, 2022. 6195 Semantic Overlap Summarization among Multiple Alternative Narratives: an Exploratory Study Naman Bansal, Mousumi Akter and Shubhra Kanti Karmaker (“Santu”) Big Data Intelligence (BDI) Lab Department of Computer Science and Software Engineering College of Engineering, Auburn University {nbansal, mza0170, sks0086}@auburn.edu Abstract In this paper, we introduce an important yet rel- atively unexplored NLP task called Semantic Overlap Summarization (SOS), which entails generating a single summary from multiple al- ternative narratives which can convey the com- mon information provided by those narratives. As no benchmark dataset is readily available for this task, we created one by collecting 2, 925 al- ternative narrative pairs from the web and then, went through the tedious process of manually creating 411 different reference summaries by engaging human annotators. As a way to evalu- ate this novel task, we first conducted a system- atic study by borrowing the popular ROUGE metric from text-summarization literature and discovered that ROUGE is not suitable for our task. Subsequently, we conducted further hu- man annotations to create 200 document-level and 1, 518 sentence-level ground-truth overlap labels. Our experiments show that the sentence- wise annotation technique with three overlap labels, i.e., {Absent (A), Partially-Present (PP), and Present (P)}, yields a higher correlation with human judgment and higher inter-rater agreement compared to the ROUGE metric. 1 Introduction In this paper, we look deeper into the challeng- ing yet relatively under-explored area of automatic summarization of multiple alternative narratives with different perspectives. To be more specific, we formally introduce a new NLP task called Semantic Overlap Summarization (SOS) from multiple alter- native narratives and conduct a systematic study of this task by creating a benchmark dataset as well as exploring how to accurately evaluate this task. SOS essentially means the task of summarizing the overlapping information present in multiple alter- nate narratives by cross-verifying their information contents against each other. Computationally, our research question is the following: Given two distinct narratives N1 and N2 of an event e, how can we automatically generate a sin- gle summary about e which conveys the common information provided by both N1 and N2? Multiple alternative narratives appear frequently in a variety of domains, including education (So- masundaran et al., 2018), the health sector (Bijoy et al., 2021), businesses intelligence (Azeroual and Theel, 2018), content analysis (Hassan et al., 2020; Karmaker Santu et al., 2018b) and privacy (Wil- son et al., 2018). Therefore, automatic summariza- tion of multiple-perspective narratives has become a pressing need in this information explosion era and can be highly useful for digesting such multi- narratives at scale and speed. Figure 1 shows a toy example of the SOS task, where both articles cover the same event related to “abortion”. However, they report from different po- litical perspectives, i.e., one from the left wing and the other from the right wing. For greater visibility, “Left” and “Right” wing reporting biases are rep- resented by blue and red text, respectively. Green …

Content truncated to 500 words