Scaling up Discourse Quality Annotation for Political Science

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 3301–3318 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 3301 Scaling up Discourse Quality Annotation for Political Science Neele Falk, Gabriella Lapesa Institute for Natural Language Processing, University of Stuttgart Pfaffenwaldring 5b 70569 Stuttgart name.surname@ims.uni-stuttgart.de Abstract The empirical quantification of the quality of a contribution to a political discussion is at the heart of deliberative theory, the subdiscipline of political science which investigates decision-making in deliberative democracy. Existing annotation on deliberative quality is time-consuming and carried out by experts, typically resulting in small datasets which also suffer from strong class imbalance. Scaling up such annotations with automatic tools is desirable, but very challenging. We take up this challenge and explore different strategies to improve the prediction of deliberative quality dimensions (justification, common good, interactivity, respect) in a standard dataset. Our results show that simple data augmentation techniques successfully alleviate data imbalance. Classifiers based on linguistic features (textual complexity and sentiment/polarity) and classifiers integrating argument quality annotations (from the argument mining community in NLP) were consistently outperformed by transformer-based models, with or without data augmentation. Keywords: annotation, evaluation, machine learning, argumentation, political science 1. Introduction How can we arrive at better political decisions? This question is at the heart of deliberative theory (Fishkin, 1995), the subfield of the political sciences which in- vestigates decision-making in deliberative democracy (e.g., in parliamentary discussions). The crucial focus of deliberative theory is on the role of the discourse ex- change that leads up to the actual decision: deliberation is not only the output of the decision making, but also the discussion that leads up to it. The basic assumption of deliberative democracy is that a rational, respectful exchange of different perspectives and arguments on an issue can lead to better decisions. This leads to the em- pirical question of measuring discourse quality. Not only is the empirical quantification of discourse qual- ity relevant to get a better theoretical understanding of the dynamics of democratic decision making, its real- life application potential is invaluable when it comes to supporting deliberation processes which take place not only in parliaments, but also in groups of lay people discussing societally relevant problems (Vecchi et al., 2021). Unsurprisingly, research in deliberative theory has es- tablished the conceptual coordinates and annotation schemas to investigate discourse quality empirically, leading to the formulation of the notion of Delibera- tive Quality. Different sub-dimensions of deliberative quality capture, for example, whether the participants in the discourse treat each other with respect, take up other opinions and sufficiently justify their positions. The best-known framework on the basis of which many discourse data is annotated is the discourse quality in- dex (DQI) (Steenbergen et al., 2003), which defines the various aspects of discourse quality. Annotation is time-consuming and requires highly trained annota- tors: it is thus no wonder that the available datasets are small. Besides, the employed annotation schemas tend to be fine-grained with multiple values for each sub- …

Content truncated to 500 words