ADePT: Auto-encoder based Differentially Private Text Transformation

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 2435–2439 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 2435 ADePT: Auto-encoder based Differentially Private Text Transformation Satyapriya Krishna Amazon Alexa satyapk@amazon.com Rahul Gupta Amazon Alexa gupra@amazon.com Christophe Dupuy Amazon Alexa dupuychr@amazon.com Abstract Privacy is an important concern when build- ing statistical models on data containing per- sonal information. Differential privacy of- fers a strong definition of privacy and can be used to solve several privacy concerns (Dwork et al., 2014). Multiple solutions have been proposed for the differentially-private transfor- mation of datasets containing sensitive infor- mation. However, such transformation algo- rithms offer poor utility in Natural Language Processing (NLP) tasks due to noise added in the process. In this paper, we address this is- sue by providing a utility-preserving differen- tially private text transformation algorithm us- ing auto-encoders. Our algorithm transforms text to offer robustness against attacks and produces transformations with high semantic quality that perform well on downstream NLP tasks. We prove the theoretical privacy guar- antee of our algorithm and assess its privacy leakage under Membership Inference Attacks (MIA) (Shokri et al., 2017) on models trained with transformed data. Our results show that the proposed model performs better against MIA attacks while offering lower to no degra- dation in the utility of the underlying trans- formation process compared to existing base- lines. 1 Introduction Differentially Private (DP) mechanisms provide ro- bustness against privacy attacks and offer practical solutions for transforming and releasing datasets without compromising privacy (Dwork et al., 2009). A typical downstream task may involve training a machine learning model with data transformed from a differentially private mechanism. However, while the DP mechanism offers privacy, it can ad- versely impact the utility of the trained model (Li and Li, 2009). Specifically, in the case of text datasets (e.g., those used in Natural Language Un- derstanding(NLU) tasks), if the DP transformation impacts the syntactic structure of the sentence or does not factor in the target NLU label (e.g. intent of the sentence in an intent classification task), the loss in utility can render the use of processed data impractical. We address this problem in the paper and introduce ADePT - an Auto encoder based Dif- ferentially Private Text transformation mechanism that process text data while reducing the impact on the utility of the dataset. The ADePT mechanism relies on text-based auto-encoders (e.g. LSTM based sequence-to- sequence models) for text transformation. An auto- encoder first transforms a given text input into some latent representation, followed by text generation (transformation) via the decoder. In this paper, we prove that the application of clipping and noising operation on the latent sentence representations returned by the encoder followed by text genera- tion by the decoder is a DP mechanism. We use ADePT to transform text datasets relevant to the Intent Classification (IC) task, where we predict in- tent of input sentence (e.g. ‘BuyTicketIntent’ intent prediction for the sentence ‘buy me a ticket to …

Content truncated to 500 words