Developing A Multilabel Corpus for the Quality Assessment of Online Political Talk

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 5503–5510 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 5503 Developing A Multilabel Corpus for the Quality Assessment of Online Political Talk Kokil Jaidka Department of Communications and New Media 11 Computing Way National University of Singapore jaidka@nus.edu.sg Abstract This paper motivates and presents the Twitter Deliberative Politics dataset, a corpus of political tweets labeled for its deliberative characteristics. The corpus was randomly sampled from replies to US congressmen and women. It is expected to be useful to a general community of computational linguists, political scientists, and social scientists interested in the study of online political expression, computer-mediated communication, and political deliberation. The data sampling and annotation methods are discussed and classic machine learning approaches are evaluated for their predictive performance on the different deliberative facets. The paper concludes with a discussion of future work aimed at developing dictio- naries for the quality assessment of online political talk in English. The dataset and a demo dashboard are available at https://github.com/kj2013/twitter-deliberative-politics. Keywords: deliberation, justification, political discussions, comments, Twitter, Facebook 1. Introduction Social media offers an ideal public sphere for citizens to engage in politics because of its inclusivity, acces- sibility, equality, and extensive outreach. Political dis- cussions on Twitter and other online platforms allow people with diverse perspectives and opinions to par- ticipate in casual conversation. However, the volume, variety, and velocity of social media posts often make hand-annotation an intractable solution to understand online political discussion dynamics. The Discourse Quality Index (Steenbergen et al., 2003) is an instru- ment to measure political discussion quality. However, so far it has been limited to the quantitative analysis of small samples of hand-annotated posts. On the other hand, a study of the political engagement, civic com- munities, echo chambers, and their dynamics requires a language quality instrument which can scale to anno- tate thousands of political social media posts. To ad- dress this research gap, this study reports on a new cor- pus of political tweets, labeled with the different facets of the discussion quality. The dataset offers a first step in building dictionaries to aid in the automatic measure- ment of the Discourse Quality Index. An emerging body of work in political communication has proposed methods that allow for automated content analysis through dictionaries (Gr¨undl, 2020), classi- fiers (Barber´a et al., 2015) and hybrid methods (Baden et al., 2020), while also discussing the possibilities and limitations of using supervised machine learning approaches as a research method (Pilny et al., 2019; Burscher et al., 2014). These methods and approaches mainly focus on frames, emotions, and incivility (Mud- diman et al., 2018) or politeness (Niculae et al., 2015). A measure of political discussion quality is scarce, and prior attempts suggest that they would not generalize well across platforms (Nelimarkka and Ahonen, 2019; Stroud et al., 2015; Halpern and Gibbs, 2013). In ad- dressing this gap, this article contributes with the first in what is to be …

Content truncated to 500 words