Selective Differential Privacy for Language Modeling

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2848 - 2859 July 10-15, 2022 ©2022 Association for Computational Linguistics Selective Differential Privacy for Language Modeling Weiyan Shi1, Aiqi Cui1, Evan Li1, Ruoxi Jia2 Zhou Yu1 Columbia University1,Virginia Tech2 {ws2634, ac4788, el3078}@columbia.edu, ruoxijia@vt.edu, , zy2461@columbia.edu Abstract With the increasing applications of language models, it has become crucial to protect these models from leaking private information. Previ- ous work has attempted to tackle this challenge by training RNN-based language models with differential privacy guarantees. However, ap- plying classical differential privacy to language models leads to poor model performance as the underlying privacy notion is over-pessimistic and provides undifferentiated protection for all tokens in the data. Given that the private in- formation in natural language is sparse (for ex- ample, the bulk of an email might not carry personally identifiable information), we pro- pose a new privacy notion, selective differential privacy, to provide rigorous privacy guarantees on the sensitive portion of the data to improve model utility. To realize such a new notion, we develop a corresponding privacy mechanism, Selective-DPSGD, for RNN-based language models. Besides language modeling, we also apply the method to a more concrete applica- tion – dialog systems. Experiments on both language modeling and dialog system build- ing show that the proposed privacy-preserving mechanism achieves better utilities while re- maining safe under various privacy attacks com- pared to the baselines. The data and code are released to facilitate future research1. 1 Introduction Language models have been widely used in vari- ous kinds of applications, such as Google Smart Compose, Amazon Alexa, and so on. However, these models are often trained on highly sensitive data such as emails and chat logs, while having the tendency to memorize the training data unintention- ally (Carlini et al., 2019, 2020). Therefore, how to protect user privacy while preserving the model utility has become an increasingly important topic. Several methods have been developed to protect the data, such as data anonymization, k-anonymity, 1https://github.com/wyshi/lm_privacy Figure 1: “Data anonymization” and training with “No DP” cannot provide knobs to adjust the trade-off be- tween privacy and utility. Selective-DP improves the privacy-utility trade-off of traditional DP, to get closer to the ideal model with both high privacy and high utility. and differential privacy (DP). Among them, DP has become a dominant privacy notion as it provides formal and provable guarantees for people to un- derstand the privacy-utility trade-off (Figure 1). It works by carefully randomizing the algorithm so that the model does not rely too much on any single data point. However, traditional DP notion protects each data point as a whole regardless of the prop- erty of individual attributes inside the data (McMa- han et al., 2018). Training large models with this overly pessimistic privacy notion could lead to poor model performance or even a non-convergent train- ing process (Kerrigan et al., 2020). Our work is inspired by an important observation that in many scenarios including …

Content truncated to 500 words