You Don’t Know My Favorite Color: Preventing Dialogue Representations from Revealing Speakers’ Private Personas

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5858 - 5870 July 10-15, 2022 ©2022 Association for Computational Linguistics You Don’t Know My Favorite Color: Preventing Dialogue Representations from Revealing Speakers’ Private Personas Haoran Li1, Yangqiu Song1, Lixin Fan2 1Dept. of CSE, Hong Kong University of Science and Technology 2AI Group, WeBank hlibt@connect.ust.hk, yqsong@cse.ust.hk, lixinfan@webank.com Abstract Social chatbots, also known as chit-chat chat- bots, evolve rapidly with large pretrained lan- guage models. Despite the huge progress, pri- vacy concerns have arisen recently: training data of large language models can be extracted via model inversion attacks. On the other hand, the datasets used for training chatbots contain many private conversations between two indi- viduals. In this work, we further investigate the privacy leakage of the hidden states of chatbots trained by language modeling which has not been well studied yet. We show that speakers’ personas can be inferred through a simple neu- ral network with high accuracy. To this end, we propose effective defense objectives to protect persona leakage from hidden states. We con- duct extensive experiments to demonstrate that our proposed defense objectives can greatly re- duce the attack accuracy from 37.6% to 0.5%. Meanwhile, the proposed objectives preserve language models’ powerful generation ability. 1 Introduction Social chatbots have been widely used to benefit many applications from answering factual ques- tions to showing emotional companionship. With recent progress in large pretrained language mod- els (Radford et al., 2019; Yang et al., 2019), some attempts (Wolf et al., 2019; Zhang et al., 2020; Ham et al., 2020; Shen et al., 2021; Sevegnani et al., 2021; Gu et al., 2021b) are made to build chatbots based on large generative language models (LMs). To train such LM-based chatbots, private conversa- tions are collected. Unfortunately, large language models tend to memorize training data and some private data can be recovered from models (Pan et al., 2020; Carlini et al., 2021). Besides such memorization problems, “overlearning” on simple training objectives can reveal sensitive attributes indirectly related to the learning task (Song and Shmatikov, 2020). LM-based social chatbots es- sentially inherit the privacy issues of general LMs and the overlearning problem. For example, as Figure 1 shows, when using a fine-tuned GPT-2 as the encoder and decoder of an LM-based social chatbot, if the learned repre- sentation of each utterance can be obtained by an adversary, then the adversary can build a classifier to predict the persona information based on the representation. As shown by the example, for five out of 14 utterances, the attacker can successfully predict the persona, which can be harmful if the users (speakers of the utterances) do not prefer to reveal the persona information. Thus, in practice, when deploying such kinds of chatbots in real ap- plications, we should first make sure that no private information can be leaked by the models. To systematically study the privacy issues in LM- based social chatbots, there are several challenges. First, there …

Content truncated to 500 words