Evaluation of Thematic Coherence in Microblogs

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 6800–6814 August 1–6, 2021. ©2021 Association for Computational Linguistics 6800 Evaluation of Thematic Coherence in Microblogs Iman Munire Bilal1 Bo Wang2,4 Maria Liakata1,3,4 Rob Procter1,4 Adam Tsakalidis3,4 1Department of Computer Science, University of Warwick 2Department of Psychiatry, University of Oxford 3School of Electronic Engineering and Computer Science, Queen Mary University of London 4The Alan Turing Institute, London, UK {iman.bilal|rob.procter}@warwick.ac.uk {mliakata|bwang|atsakalidis}@turing.ac.uk Abstract Collecting together microblogs representing opinions about the same topics within the same timeframe is useful to a number of dif- ferent tasks and practitioners. A major ques- tion is how to evaluate the quality of such the- matic clusters. Here we create a corpus of mi- croblog clusters from three different domains and time windows and define the task of eval- uating thematic coherence. We provide an- notation guidelines and human annotations of thematic coherence by journalist experts. We subsequently investigate the efficacy of differ- ent automated evaluation metrics for the task. We consider a range of metrics including sur- face level metrics, ones for topic model co- herence and text generation metrics (TGMs). While surface level metrics perform well, out- performing topic coherence metrics, they are not as consistent as TGMs. TGMs are more re- liable than all other metrics considered for cap- turing thematic coherence in microblog clus- ters due to being less sensitive to the effect of time windows. 1 Introduction As social media gains popularity for news track- ing, unfolding stories are accompanied by a vast spectrum of reactions from users of social media platforms. Topic modelling and clustering methods have emerged as potential solutions to challenges of filtering and making sense of large volumes of mi- croblog posts (Rosa et al., 2011; Aiello et al., 2013; Resnik et al., 2015; Surian et al., 2016). Providing a way to access easily a wide range of reactions around a topic or event has the potential to help those, such as journalists (Tolmie et al., 2017), police (Procter et al., 2013), health (Furini and Menegoni, 2018) and public safety professionals (Procter et al., 2020), who increasingly rely on so- cial media to detect and monitor progress of events, public opinion and spread of misinformation. Recent work on grouping together views about tweets expressing opinions about the same enti- ties has obtained clusters of tweets by leveraging two topic models in a hierarchical approach (Wang et al., 2017b). The theme of such clusters can either be represented by their top-N highest-probability words or measured by the semantic similarity among the tweets. One of the questions regard- ing thematic clusters is how well the posts grouped together relate to each other (thematic coherence) and how useful such clusters can be. For example, the clusters can be used to discover topics that have low coverage in traditional news media (Zhao et al., 2011). Wang et al. (2017a) employ the centroids of Twitter clusters as the basis for …

Content truncated to 500 words