Improving Neural Political Statement Classification with Class Hierarchical Information

Findings of the Association for Computational Linguistics: ACL 2022, pages 2367 - 2382 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Improving Neural Political Statement Classification with Class Hierarchical Information Erenay Dayanik1, André Blessing1, Nico Blokker2, Sebastian Haunss2, Jonas Kuhn1, Gabriella Lapesa1, and Sebastian Padó1 1IMS, University of Stuttgart, Germany 2SOCIUM, University of Bremen, Germany Abstract Many tasks in text-based computational social science (CSS) involve the classification of po- litical statements into categories based on a domain-specific codebook. In order to be use- ful for CSS analysis, these categories must be fine-grained. The typically skewed distri- bution of fine-grained categories, however, re- sults in a challenging classification problem on the NLP side. This paper proposes to make use of the hierarchical relations among cat- egories typically present in such codebooks: e.g., markets and taxation are both subcate- gories of economy, while borders is a subcat- egory of security. We use these ontological relations as prior knowledge to establish addi- tional constraints on the learned model, thus improving performance overall and in particu- lar for infrequent categories. We evaluate sev- eral lightweight variants of this intuition by ex- tending state-of-the-art transformer-based text classifiers on two datasets and multiple lan- guages. We find the most consistent improve- ment for an approach based on regularization. 1 Introduction The argumentative or discursive turn in policy anal- ysis and political science more generally has long established the value of textual sources for the anal- ysis of politics and policies (Fischer and Forester, 1993). Traditionally, data sources such as inter- views or newspaper reports were annotated using various methods of qualitative text analysis (Wa- genaar, 2011; Mayring, 2019). At the heart of this analysis is always a codebook, i.e., guidelines that map actual statements or textual passages to the ab- stract concepts relevant for the respective research. Categories in codebooks are almost always ar- ranged hierarchically, with fine-grained categories being grouped together into supercategories that are often, but not always, more abstract. Fine- grained categories are generally generated induc- tively from the analyzed texts in an iterative pro- cess of summarizing and abstracting from the orig- inal text, while the supercategories are deductively generated from existing knowledge of the relevant policy field and from theoretical and conceptual findings of prior research. For example, the code- book of the long-running Comparative Manifesto Project (CMP), which analyzes party manifestos across several countries, includes 7 supercategories (such as external relations or economy) with 56 subcategories: for economy, among others, free market, market regulation, economic goals, etc. (Merz et al., 2016; Werner et al., 2011). Here, supercategories represent the separation of policy fields that is reflected in political institutions, e.g., ministries. Fine-grained, hierarchical schemes help researchers both with data annotation and with anal- ysis. Annotation is often easier when the annotation decision is (implicitly) first based on a supercate- gory and then on fine-grained subcategories. For analysis, supercategories structure the annotated material according to different levels of abstraction, thereby supporting interpretation and modeling. While such a hierarchical process a natural …

Content truncated to 500 words