Probing Power by Prompting: Harnessing Pre-trained Language Models for Power Connotation Framing

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 873–885 May 2-6, 2023 ©2023 Association for Computational Linguistics Probing Power by Prompting: Harnessing Pre-trained Language Models for Power Connotation Framing Shima Khanehzar† Trevor Cohn† Gosia Mikolajczak� Lea Frermann† † School of Computing and Information Systems, The University of Melbourne � The Global Institute for Women’s Leadership, The Australian National University skhanehzar@student.unimelb.edu.au {tcohn,lfrermann}@unimelb.edu.au Gosia.Mikolajczak@anu.edu.au Abstract Subtle changes in word choice in communi- cation can evoke very different associations with the involved actors. For instance, a com- pany ‘employing workers’ evokes a more posi- tive connotation than the one ‘exploiting’ them. This concept is called connotation. This pa- per investigates whether pre-trained language models (PLMs) encode such subtle connotative information about power differentials between involved entities. We design a probing frame- work for power connotation, building on Sap et al. (2017)’s operationalization of connota- tion frames. We show that zero-shot prompting of PLMs leads to above chance prediction of power connotation, however fine-tuning PLMs using our framework drastically improves their accuracy. Using our fine-tuned models, we present a case study of power dynamics in US news reporting on immigration, showing the potential of our framework as a tool for under- standing subtle bias in the media.1 1 Introduction An author’s choice of words evokes associations and reactions from a reader that go beyond the lit- eral meaning they express. This underlying level of meaning is called connotation and often carries social, cultural, or emotional implications for lis- teners or readers (Sonesson, 1998). In high-stakes settings such as opinion polls or news reporting, it has been shown that subtle changes in word choices can influence responses or opinions (Kahneman and Tversky, 1984; Rashkin et al., 2016). For ex- ample, the choice of the term ‘undocumented work- ers’ vs. ‘illegal aliens’ to describe immigrants can elicit different levels of prejudice toward that group (M.S., 2010). Different connotations can inject multiple layers of meaning into a word, phrase, or passage. This 1Source code is available at https://github.com/shinyemimalef/Probing-Power-by- Prompting Prelude The crowd approves the leader. Hypothesis The leader has MASK power than the crowd. Connotation AG≻powerTH Prelude The crowd hails the leader. Hypothesis The leader has MASK power than the crowd. Connotation AG≺powerTH Figure 1: Top: Connotation frames from Sap et al. (2017)’s dataset for the predicates ‘approve’ and ‘hail’, which evoke different power differentials between argu- ments ‘crowd’ and ‘leader’. Predicate ‘approve’ implies that the leader needs approval of the powerful crowd; while ‘hail’ suggests that the crowd praises the pow- erful leader. Colored arrows show how we map from predicate- to entity-centric connotation. Bottom: Our PLM probing framework comprising a prelude (intro- ducing predicate and arguments) and a hypothesis (ex- pressing the implied power differential). paper focuses on connotations of power arising from descriptions of entities. Power dynamics are omnipresent at all levels of society, between indi- viduals, groups, political actors, or institutions, and the subjective power of an entity can be expressed through the choice of …

Content truncated to 500 words