Masking Actor Information Leads to Fairer Political Claims Detection

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4385–4391 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 4385 Masking Actor Information Leads to Fairer Political Claims Detection Erenay Dayanik and Sebastian Pad´o IMS, University of Stuttgart Stuttgart, Germany {erenay.dayanik,sebastian.pado}@ims.uni-stuttgart.de Abstract A central concern in Computational Social Sci- ences (CSS) is fairness: where the role of NLP is to scale up text analysis to large corpora, the quality of automatic analyses should be as independent as possible of textual properties. We analyze the performance of a state-of-the- art neural model on the task of political claims detection (i.e., the identification of forward- looking statements made by political actors) and identify a strong frequency bias: claims made by frequent actors are recognized bet- ter. We propose two simple debiasing methods which mask proper names and pronouns dur- ing training of the model, thus removing per- sonal information bias. We find that (a) these methods significantly decrease frequency bias while keeping the overall performance stable; and (b) the resulting models improve when evaluated in an out-of-domain setting. 1 Introduction In recent years, NLP methods have found increas- ing adoption in the social sciences as part of the movement towards Computational Social Sciences or CSS (Lazer et al., 2009). An important part of the appeal of CSS is the promise to scale up the amount of data under consideration: from what can be annotated manually to what can be analyzed au- tomatically, typically an increase by several orders of magnitude, enabling a paradigm shift towards new research questions (Chang et al., 2014). How- ever, this shift comes with new challenges: if the analyses are carried out by a machine, how can we trust that any outcomes really stem from the under- lying data, rather than from processing artifacts? Consequently, CSS must be crucially interested in the algorithmic fairness or (absence of) bias of the underlying machine learning methods (e.g., Binns, 2018; Canetti et al., 2019). However, work on this topic in NLP over the last years has found Angela Merkel called for swift tax cuts. Actor Claim Support During yesterday’s cabinet meeting in Berlin, Figure 1: Political claims detection: Text (above), actor–polarity–claim structure (below) that more applications contain biases than not, in- cluding lexical semantics (Bolukbasi et al., 2016), emotion detection (Kiritchenko and Mohammad, 2018), coreference (Zhao et al., 2018), recommen- dation generation (Chakraborty et al., 2019) and textual inference (Rudinger et al., 2017). It is there- fore surprising that, to our knowledge, the bias of NLP methods applied in the CSS domain have found little attention so far. In this paper, we consider the CSS task of politi- cal claim analysis (Koopmans and Statham, 2010), an entity and relation extraction task from the do- main of argument(ation) mining (Cabrio and Vil- lata, 2018). Its goal is to extract (Actor, Polarity, Claim) tuples from text, as illustrated in Figure 1. This is a structured prediction task with the goal of identifying actors, their claims, and polarities (sup- …

Content truncated to 500 words