Provably Confidential Language Modelling

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 943 - 955 July 10-15, 2022 ©2022 Association for Computational Linguistics Provably Confidential Language Modelling Xuandong Zhao Lei Li Yu-Xiang Wang University of California, Santa Barbara {xuandongzhao,leili,yuxiangw}@cs.ucsb.edu Abstract Large language models are shown to memo- rize privacy information such as social secu- rity numbers in training data. Given the sheer scale of the training corpus, it is challenging to screen and filter all privacy data, either man- ually or automatically. In this paper, we pro- pose Confidentially Redacted Training (CRT), a method to train language generation models while protecting the confidential segments. We borrow ideas from differential privacy (which solves a related but distinct problem) and show that our method is able to provably prevent un- intended memorization by randomizing parts of the training process. Moreover, we show that redaction with an approximately correct screening policy amplifies the confidentiality guarantee. We implement the method for both LSTM and GPT language models. Our experi- mental results show that the models trained by CRT obtain almost the same perplexity while preserving strong confidentiality1. 1 Introduction Language models (LM) have rich real-world ap- plications in, among others, machine translation (Bahdanau et al., 2015), AI chatbots (Hosseini-Asl et al., 2020), question answering (Kwiatkowski et al., 2019), and information retrieval (Ganguly et al., 2015). The advent of transformers (Vaswani et al., 2017) has fostered a dramatic advancement in the capabilities of generative neural language models, yet they come at a cost to privacy, as the amount of excess parameters in the LM enables it to memorize certain training samples. Recent works show that sensitive user information from the training dataset, such as address and name, can be extracted verbatim from text generation mod- els by querying the LM as an API (Carlini et al., 2019, 2021; Lee et al., 2022). How to train a high- performing language model without memorizing 1Our code is available at https://github.com/ XuandongZhao/CRT sensitive text has become a major research chal- lenge. Existing solutions to this problem primarily leverage differential privacy (DP) (Dwork et al., 2006). Differentially private learning algorithms ensure that an attacker could not infer whether a data point is used for training, let alone extracting the sensi- tive information within that data point. However, there are several mismatches between the problem of privacy that DP addresses, and our problem of preventing the memorization of sensi- tive text (henceforth referred to as confidentiality). First, confidential information in a natural language dataset is sparse (e.g., the bulk of an email might not carry confidential information). DP’s undis- criminating protection for all sentences could be unnecessarily conservative which limits the util- ity of the trained model. Second, what needs to be protected is the content of the sensitive text, rather than the data context. For example, in the sentence “My SSN is 123-45-6789.”, it is the actual SSN that we hope to conceal rather than the general information that someone entered …

Content truncated to 500 words