Privacy-Preserving Text Classification on BERT Embeddings with Homomorphic Encryption

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3169 - 3175 July 10-15, 2022 ©2022 Association for Computational Linguistics Privacy-Preserving Text Classification on BERT Embeddings with Homomorphic Encryption Garam Lee*1 Minsoo Kim*2 Jai Hyun Park*2 Seung-won Hwang†2 Jung Hee Cheon1,2 1CryptoLab 2Seoul National University garamlee@cryptolab.co.kr {minsoo9574, jhyunp, seungwonh, jhcheon}@snu.ac.kr Abstract Embeddings, which compress information in raw text into semantics-preserving low- dimensional vectors, have been widely adopted for their efficacy. However, recent research has shown that embeddings can potentially leak pri- vate information about sensitive attributes of the text, and in some cases, can be inverted to recover the original input text. To address these growing privacy challenges, we propose a pri- vatization mechanism for embeddings based on homomorphic encryption, to prevent po- tential leakage of any piece of information in the process of text classification. In partic- ular, our method performs text classification on the encryption of embeddings from state- of-the-art models like BERT, supported by an efficient GPU implementation of CKKS en- cryption scheme. We show that our method offers encrypted protection of BERT embed- dings, while largely preserving their utility on downstream text classification tasks. 1 Introduction In recent years, the increasingly wide adoption of vector-based representations of text such as BERT, eLMo, and GPT (Devlin et al., 2019; Peters et al., 2018; Radford et al., 2019), has called attention to the privacy ramifications of embedding mod- els. For example, Coavoux et al. (2018); Li et al. (2018) show that sensitive information such as the authors’ gender and age can be partially recovered from an embedded representation of text. Song and Raghunathan (2020) report that BERT-based sentence embeddings can be inverted to recover up to 50%–70% of the input words. Previously proposed solutions such as differen- tial privacy based on perturbation/noise (Qu et al., 2021), require manually controlling the noise in- jected into embeddings, to control the privacy- utility trade-off to a level suitable for each down- stream task. In this work, we propose a privacy *Equal contribution. †Corresponding author. solution based on Approximate Homomorphic En- cryption, which is able to achieve little to no accu- racy loss of BERT embeddings on text classifica- tion1, while ensuring a desired level of encrypted protection, i.e. 128-bit security. Homomorphic Encryption (HE) is a crypto- graphic primitive that serves computations over en- crypted data without any decryption process. While previous works have focused on homomorphic computation where the inputs are numerical data, in applications such as privacy-preserving machine learning algorithms (Lauter, 2021), logistic regres- sion (Kim et al., 2018), and neural network infer- ence (Gilad-Bachrach et al., 2016), they have rarely been applied to unstructured data such as text. Re- cent works in this direction include Podschwadt and Takabi (2020), who conduct sentiment clas- sification over encrypted word embeddings using RNN. However, they use a simple embedding layer which maps words in a dictionary to real-valued vectors, and model training is only supported on plaintext. The most closely related …

Content truncated to 500 words