THE-X: Privacy-Preserving Transformer Inference with Homomorphic Encryption

Findings of the Association for Computational Linguistics: ACL 2022, pages 3510 - 3520 May 22-27, 2022 c⃝2022 Association for Computational Linguistics THE-X: Privacy-Preserving Transformer Inference with Homomorphic Encryption Tianyu Chenp o ⋆, Hangbo Baon, Shaohan Huangn, Li Dongn, Binxing Jiaom, Daxin Jiangm, Haoyi Zhoup o, Jianxin Lip o u, Furu Wein BDBC, Beihang University, Chinap SKLSDE, Beihang University, Chinao Microsoft Researchn NLP Group, Microsoft STCAm {tianyuc, zhouhy,lijx}@buaa.edu.cn {t-habao,shaohanh, lidong1, binxjia, djiang, fuwei}@microsoft.com Abstract As more and more pre-trained language mod- els adopt on-cloud deployment, the privacy is- sues grow quickly, mainly for the exposure of plain-text user data (e.g., search history, medi- cal record, bank account). Privacy-preserving inference of transformer models is on the de- mand of cloud service users. To protect pri- vacy, it is an attractive choice to compute only with ciphertext in homomorphic encryp- tion (HE). However, enabling pre-trained mod- els inference on ciphertext data is difficult due to the complex computations in transformer blocks, which are not supported by current HE tools yet. In this work, we introduce THE-X, an approximation approach for transformers, which enables privacy-preserving inference of pre-trained models developed by popular frameworks. THE-X proposes a workflow to deal with complex computation in transformer networks, including all the non-polynomial functions like GELU, softmax, and Layer- Norm. Experiments reveal our proposed THE- X can enable transformer inference on en- crypted data for different downstream tasks, all with negligible performance drop but enjoy- ing the theory-guaranteed privacy-preserving advantage. 1 Introduction Accompanying the revolution of pre-trained mod- els in many NLP applications, such as senti- ment analysis (Xu et al., 2019a), question an- swering (Yang et al., 2019b), information re- trieval (Yang et al., 2019c), and text genera- tion (Raffel et al., 2020), many related technologies have been deployed on the cloud to process user data from personal customers, small businesses, and large enterprises by industrial service providers. However, the convenience of the on-cloud pre- training technology also comes with a series of ⋆ Contribution during internship at MSRA. uThe corre- sponding author is Jianxin Li <lijx@buaa.edu.cn>. Finetuned Model Cloud Service （encrypted text) Encrypt (data, key) Decrypt (res, key) User (plain text) Figure 1: An overview of our THE-X. The transformer- based model could inference on encrypted data with our THE-X, enabling theory-guaranteed privacy protec- tion for users. privacy challenges due to the sensitive nature of user data. For example, the input text or even text vector representations in user requests can leak pri- vate information, which may cause the specific user to be identified (Schwartz and Solove, 2011; Zhu and Han, 2020). This lack of privacy guarantees may impede privacy-conscious users from releas- ing their data to service providers. Thus, service providers may suffer from the deficiency of evolv- ing models with user data. Besides, unintended data disclosure and other privacy breaches may result in litigation, fines, and reputation damages for service providers. These concerns spark our proposal of THE-X, to enable privacy-preserving inference of transformer. Specifically, we identify two challenges for the privacy-preserving inference of …

Content truncated to 500 words