Extracted BERT Model Leaks More Information than You Think!

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1530 - 1537 December 7-11, 2022 ©2022 Association for Computational Linguistics Extracted BERT Model Leaks More Information than You Think! Xuanli He1∗, Chen Chen2∗, Lingjuan Lyu3†, Qiongkai Xu4 1University College London, 2Zhejiang University, 3Sony AI, 4 The University of Melbourne zodiac.he@gmail.com, cc33@zju.edu.cn Lingjuan.Lv@sony.com, Qiongkai.Xu@unimelb.edu.au Abstract The collection and availability of big data, combined with advances in pre-trained models (e.g. BERT), have revolutionized the predic- tive performance of natural language process- ing tasks. This allows corporations to provide machine learning as a service (MLaaS) by en- capsulating fine-tuned BERT-based models as APIs. Due to significant commercial interest, there has been a surge of attempts to steal re- mote services via model extraction. Although previous works have made progress in defend- ing against model extraction attacks, there has been little discussion on their performance in preventing privacy leakage. This work bridges this gap by launching an attribute inference at- tack against the extracted BERT model. Our extensive experiments reveal that model ex- traction can cause severe privacy leakage even when victim models are facilitated with ad- vanced defensive strategies. 1 Introduction The emergence of pre-trained language models (PLMs) has revolutionized the natural language pro- cessing (NLP) research, leading to state-of-the-art (SOTA) performance on a wide range of tasks (De- vlin et al., 2018; Yang et al., 2019). This break- through has enabled commercial companies to de- ploy machine learning models as black-box APIs on their cloud platforms to serve millions of users, such as Google Prediction API1, Microsoft Azure Machine Learning2, and Amazon Machine Learn- ing3. However, recent works have shown that existing NLP APIs are vulnerable to model extraction attack (MEA), which can reconstruct a copy of the remote ∗Equal contribution. Most of the work was finished when X.H. was at Monash University. Work done during C.C.’s internship at Sony AI. †Corresponding author. 1https://cloud.google.com/prediction 2https://studio.azureml.net 3https://aws.amazon.com/machine-learning NLP model based on the carefully-designed queries and outputs of the target API (Krishna et al., 2019; Wallace et al., 2020), causing the financial losses of the target API. Prior to our work, researchers have investigated the hazards of model extraction under various settings, including stealing commer- cial APIs (Wallace et al., 2020; Xu et al., 2022), ensemble model extraction (Xu et al., 2022), and adversarial examples transfer (Wallace et al., 2020; He et al., 2021). Previous works have indicated that an adver- sary can leverage the extracted model to conduct adversarial example transfer, such that these ex- amples can corrupt the predictions of the victim model (Wallace et al., 2020; He et al., 2021). Given the success of MEA and adversarial example trans- fer, we conjecture that the predictions from a vic- tim model could reveal its private information un- consciously, as victim models can memorize side information in addition to the task-related mes- sage (Lyu and Chen, 2020; Lyu et al., 2020; Carlini et al., 2021). Thus, we are interested in examining whether the victim model can leak the private …

Content truncated to 500 words