Just Fine-tune Twice: Selective Differential Privacy for Large Language Models

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6327 - 6340 December 7-11, 2022 ©2022 Association for Computational Linguistics Just Fine-tune Twice: Selective Differential Privacy for Large Language Models Weiyan Shi†, Ryan Shea†, Si Chen‡, Chiyuan Zhang⋄, Ruoxi Jia‡, Zhou Yu† Columbia University†, Virginia Tech‡, Google Research⋄ {ws2634,rs4235}@columbia.edu, chensi@vt.edu, chiyuan@google.com, ruoxijia@vt.edu, zy2461@columbia.edu Abstract Protecting large language models from privacy leakage is becoming increasingly crucial with their wide adoption in real-world products. Yet applying differential privacy (DP), a canonical notion with provable privacy guarantees for ma- chine learning models, to those models remains challenging due to the trade-off between model utility and privacy loss. Utilizing the fact that sensitive information in language data tends to be sparse, Shi et al. (2021) formalized a DP no- tion extension called Selective Differential Pri- vacy (SDP) to protect only the sensitive tokens defined by a policy function. However, their al- gorithm only works for RNN-based models. In this paper, we develop a novel framework, Just Fine-tune Twice (JFT), that achieves SDP for state-of-the-art large transformer-based models. Our method is easy to implement: it first fine- tunes the model with redacted in-domain data, and then fine-tunes it again with the original in-domain data using a private training mecha- nism. Furthermore, we study the scenario of im- perfect implementation of policy functions that misses sensitive tokens and develop systematic methods to handle it. Experiments show that our method achieves strong utility compared to previous baselines. We also analyze the SDP privacy guarantee empirically with the canary insertion attack1. 1 Introduction With the rapid advancement in natural language processing (NLP), it has become increasingly im- portant to protect NLP models from leaking privacy information. Previous work has attempted to tackle this challenge by applying differential privacy (DP, Dwork et al., 2014) on these models (McMahan et al., 2018; Li et al., 2021). However, existing DP learning algorithms suffer from limited user control and low utility, as they protect the entirety 1Our code and data are available at https://github. com/wyshi/sdp_transformers of each training example (e.g., one complete sen- tence) regardless of users’ privacy preference, and therefore tend to be overly pessimistic when only partial information in a training example is sensi- tive. This problem is particularly pertinent in NLP, as NLP training data are often mixed with sparse domain-dependent private information, and not all tokens need to be protected. For example, for the sentence “My SSN is 123-45-6789”, only the last few tokens of the actual SSN need to be protected. In fact, the definition of DP does not prevent us at all from protecting only the sensitive part of data. Specifically, DP ensures that the output of a data analysis algorithm stays roughly the same for neighboring datasets, while providing the flexibil- ity to adjust the definition of neighboring relation to specific application contexts. Shi et al. (2021) recently proposed an instantiation of DP, called Selective-DP (SDP), which defines neighboring datasets to differ only in the sensitive part of a training example …

Content truncated to 500 words