Analyzing the Dialect Diversity in Multi-document Summaries

Proceedings of the 29th International Conference on Computational Linguistics, pages 6208–6221 October 12–17, 2022. 6208 Analyzing the Dialect Diversity in Multi-document Summaries Olubusayo Olabisi, Aaron Hudson, Antonie Jetter, Ameeta Agrawal Portland State University {oolabisi,ahuds2,ajetter,ameeta}@pdx.edu Abstract Social media posts provide a compelling, yet challenging source of data of diverse perspec- tives from many socially salient groups. Au- tomatic text summarization algorithms make this data accessible at scale by compressing large collections of documents into short sum- maries that preserve salient information from the source text. In this work, we take a comple- mentary approach to analyzing and improving the quality of summaries generated from social media data in terms of their ability to represent salient as well as diverse perspectives. We in- troduce a novel dataset, DivSumm, of dialect di- verse tweets and human-written extractive and abstractive summaries1. Then, we study the extent of dialect diversity reflected in human- written reference summaries as well as system- generated summaries. The results of our ex- tensive experiments suggest that humans anno- tate fairly well-balanced dialect diverse sum- maries, and that cluster-based pre-processing approaches seem beneficial in improving the overall quality of the system-generated sum- maries without loss in diversity. 1 Introduction Since the launch of Twitter, its short, informal, creative, albeit noisy, social media posts called tweets, have been collected, labeled, and studied in numerous natural language processing (NLP) tasks, ranging from identifying topics and places of rising interest to sentiment analysis, and more. These easily accessible user-generated tweets, produced contemporaneously as world and private events un- fold, provide insights into the perspective of diverse social groups but are too manifold for humans to interpret at scale. In response, automatic text sum- marization algorithms aim to compress long pieces of text into short, fluent, and consistent summaries while preserving the most salient information from 1DivSumm dataset is available at https://github. com/PortNLP/DivSumm the source text (Meng et al., 2012; Lin et al., 2021a; Amplayo et al., 2021). Like other NLP models, summarization algo- rithms run the risk of perpetuating unintentional social biases against diverse groups (e.g., race or gender) and promoting structures and practices that systematically limit some groups’ access to resources and decision-making power (Blodgett et al., 2020). This is because the collections of online texts such as news or Wikipedia articles, typically used for developing NLP algorithms, re- flect the interests, language patterns, and structured writing style of their author demographics, which differ from those of other communities. Algorithms trained on such datasets may produce synopses in which diverse perspectives are systematically ex- cluded. This means that groups who manage to overcome existing barriers to participation, for in- stance, via social media posts, and who speak up and offer their perspectives may still not be heard. Fairness definitions for NLP models are gen- erally based on the notion of equal treatment – an algorithm is considered fair when it performs the same for mainstream and underrepresented groups (i.e., group fairness) or delivers the same conclusions about an individual, regardless of the group …

Content truncated to 500 words