Parameter-efficient Modularised Bias Mitigation via AdapterFusion

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2738–2751 May 2-6, 2023 ©2023 Association for Computational Linguistics Parameter-efficient Modularised Bias Mitigation via AdapterFusion Deepak Kumar1, Oleg Lesota1, George Zerveas3, Daniel Cohen3,4, Carsten Eickhoff3,5, Markus Schedl1,2, Navid Rekabsaz1,2 1Johannes Kepler University Linz, Austria 2Linz Institute of Technology, AI Lab {deepak.kumar,oleg.lesota,markus.schedl,navid.rekabsaz}@jku.at 3AI Lab, Brown University, USA george_zerveas@brown.edu 4Dataminr, USA daniel.cohen@dataminr.com 5University of Tübingen, Germany c.eickhoff@acm.org Abstract Large pre-trained language models contain so- cietal biases and carry along these biases to downstream tasks. Current in-processing bias mitigation approaches (like adversarial train- ing) impose debiasing by updating a model’s parameters, effectively transferring the model to a new, irreversible debiased state. In this work, we propose a novel approach to develop stand-alone debiasing functionalities separate from the model, which can be integrated into the model on-demand, while keeping the core model untouched. Drawing from the concept of AdapterFusion in multi-task learning, we introduce DAM (Debiasing with Adapter Mod- ules) – a debiasing approach to first encapsu- late arbitrary bias mitigation functionalities into separate adapters, and then add them to the model on-demand in order to deliver fairness qualities. We conduct a large set of experi- ments on three classification tasks with gender, race, and age as protected attributes. Our re- sults show that DAM improves or maintains the effectiveness of bias mitigation, avoids catas- trophic forgetting in a multi-attribute scenario, and maintains on-par task performance, while granting parameter-efficiency and easy switch- ing between the original and debiased models. 1 Introduction Large pre-trained language models (PLM) and their variations fine-tuned on downstream tasks encode societal biases and stereotypes. A large body of research shows the existence of these biases (Zhao et al., 2019; Sheng et al., 2019; Rekabsaz et al., 2021b), and discuss their potential harms (Blod- gett et al., 2020) in various tasks such as occupa- tion prediction from biographies (De-Arteaga et al., 2019), information retrieval (IR) (Rekabsaz et al., 2021a; Rekabsaz and Schedl, 2020), and machine translation (Stanovsky et al., 2019). A common ap- proach to mitigating these biases is to adjust model parameters according to specific fairness criteria, Task Adapter Add & Norm Multi-Head Attention Debiasing Adapter Add & Norm Add & Norm Feed Forward AdapterFusion Figure 1: A transformer block in DAM to debias arbi- trarily many protected attributes in different modules. achieved using optimization methods such as adver- sarial training (Elazar and Goldberg, 2018; Barrett et al., 2019; Han et al., 2021b; Rekabsaz et al., 2021a). The resulting debiased models significantly di- minish the effect of protected attributes (e. g. gen- der, race, etc.) on model’s predictions. However, depending on the usage context, characteristics of the input or fairness-utility trade-off considerations, system designers or end-users might in practice still prefer to instead use the original (potentially biased) model for processing some requests. As an example, while a fairness-aware document retrieval model should tease out gender attributes when pro- cessing bias-sensitive queries (Krieg et al., 2022) (like how to become CEO?), other common queries …

Content truncated to 500 words