When differential privacy meets NLP: The devil is in the detail

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1522–1528 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 1522 When differential privacy meets NLP: The devil is in the detail Ivan Habernal Trustworthy Human Language Technologies Department of Computer Science Technical University of Darmstadt ivan.habernal@tu-darmstadt.de www.trusthlt.org Abstract Differential privacy provides a formal ap- proach to privacy of individuals. Applica- tions of differential privacy in various scenar- ios, such as protecting users’ original utter- ances, must satisfy certain mathematical prop- erties. Our contribution is a formal analy- sis of ADePT, a differentially private auto- encoder for text rewriting (Krishna et al., 2021). ADePT achieves promising results on downstream tasks while providing tight privacy guarantees. Our proof reveals that ADePT is not differentially private, thus ren- dering the experimental results unsubstanti- ated. We also quantify the impact of the error in its private mechanism, showing that the true sensitivity is higher by at least factor 6 in an op- timistic case of a very small encoder’s dimen- sion and that the amount of utterances that are not privatized could easily reach 100% of the entire dataset. Our intention is neither to crit- icize the authors, nor the peer-reviewing pro- cess, but rather point out that if differential pri- vacy applications in NLP rely on formal guar- antees, these should be outlined in full and put under detailed scrutiny. 1 Introduction The need for NLP systems to protect individuals’ privacy has led to the adoption of differential pri- vacy (DP). DP methods formally guarantee that the output of the algorithm will be ‘roughly the same’ regardless of whether or not any single individual is present in the central dataset; this is achieved by employing randomized algorithms (Dwork and Roth, 2013). Local DP, a variant of DP, mitigates the need for a central dataset and applies random- ization on each individual’s datapoint. Local DP thus guarantees that its output for an individual A will be ‘almost indistinguishable’ from the output of any other individuals B or C.1 1See the randomized response for an easy explanation of local DP for a single bit (Warner, 1965). This level of privacy protection makes local DP an ideal framework for NLP applications that op- erate on sensitive user input which should not be collected and processed globally by an untrusted party, e.g., users’ verbatim utterances. When the utterances are ‘privatized’ by local DP, any future post-processing or adversarial attack cannot reveal more than allowed by the particular local DP al- gorithm’s properties (namely the ε parameter; see later Sec. 2). ADePT, a local DP algorithm recently published at EACL by Krishna et al. (2021) from Ama- zon Alexa, proposed a differentially private auto- encoder for text rewriting. In summary, ADePT takes an input textual utterance and re-writes it in a way such that the output satisfies local DP guar- antees. Unfortunately, a thorough formal analysis reveals that ADePT is in fact not differentially pri- vate and the privatized data do not protect privacy of …

Content truncated to 500 words