Prompt Tuning Pushes Farther, Contrastive Learning Pulls Closer: A Two-Stage Approach to Mitigate Social Biases

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 14254–14267 July 9-14, 2023 ©2023 Association for Computational Linguistics Prompt Tuning Pushes Farther, Contrastive Learning Pulls Closer: A Two-Stage Approach to Mitigate Social Biases Yingji Li1, Mengnan Du2, Xin Wang3∗, Ying Wang1,4∗ 1College of Computer Science and Technology, Jilin University, Changchun, China 2Department of Data Science, New Jersey Institute of Technology, Newark, USA 3School of Artificial Intelligence, Jilin University, Changchun, China 4Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University, Changchun, China yingji21@mails.jlu.edu.cn, mengnan.du@njit.edu, {xinwang,wangying2010}@jlu.edu.cn Abstract As the representation capability of Pre-trained Language Models (PLMs) improve, there is growing concern that they will inherit social biases from unprocessed corpora. Most pre- vious debiasing techniques used Counterfac- tual Data Augmentation (CDA) to balance the training corpus. However, CDA slightly modi- fies the original corpus, limiting the representa- tion distance between different demographic groups to a narrow range. As a result, the debiasing model easily fits the differences be- tween counterfactual pairs, which affects its debiasing performance with limited text re- sources. In this paper, we propose an adversar- ial training-inspired two-stage debiasing model using Contrastive learning with Continuous Prompt Augmentation (named CCPA) to mit- igate social biases in PLMs’ encoding. In the first stage, we propose a data augmenta- tion method based on continuous prompt tun- ing to push farther the representation distance between sample pairs along different demo- graphic groups. In the second stage, we utilize contrastive learning to pull closer the represen- tation distance between the augmented sample pairs and then fine-tune PLMs’ parameters to get debiased encoding. Our approach guides the model to achieve stronger debiasing perfor- mance by adding difficulty to the training pro- cess. Extensive experiments show that CCPA outperforms baselines in terms of debiasing per- formance. Meanwhile, experimental results on the GLUE benchmark show that CCPA retains the language modeling capability of PLMs. 1 Introduction Pre-trained Language Models (PLMs) have demon- strated outstanding performance in recent years and have been widely used in natural language under- standing tasks (Peters et al., 2018; Delobelle et al., 2022). However, the powerful language modeling ∗Corresponding author PLM Unaugmented positive sample pair New positive sample pair Contrastive training PLM’ Augmented positive sample pair Contrastive training Figure 1: The motivation of CCPA. For the new input samples, the PLM’s performance of the augmented sam- ple training is stronger than that of the unaugmented sample training. capability enables PLMs to learn good representa- tions from large-scale training corpora while captur- ing human-like social biases. Recent studies have demonstrated that the representations encoded by PLMs learn social biases specific to demographic groups (e.g., gender, race, religion) and can be amplified to downstream tasks, leading to unfair outcomes and adverse social effects (Zhao et al., 2019; Webster et al., 2020). As a result, mitigating social biases in PLMs’ encoding can improve the fairness of NLP systems significantly (Bolukbasi et al., 2016; Bender and Friedman, 2018). Most existing debiasing techniques first need …

Content truncated to 500 words