Are You Copying My Model? Protecting the Copyright of Large Language Models for EaaS via Backdoor Watermark

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 7653–7668 July 9-14, 2023 ©2023 Association for Computational Linguistics Are You Copying My Model? Protecting the Copyright of Large Language Models for EaaS via Backdoor Watermark Wenjun Peng1∗, Jingwei Yi1∗, Fangzhao Wu2†, Shangxi Wu3, Bin Zhu2, Lingjuan Lyu4, Binxing Jiao5, Tong Xu1†, Guangzhong Sun1, Xing Xie2 1University of Science and Technology of China 2Microsoft Research Asia 3Beijing Jiaotong University 4Sony AI 5Microsoft STC Asia {pengwj,yjw1029}@mail.ustc.edu.cn wufangzhao@gmail.com wushangxi@bjtu.edu.cn {binzhu,binxjia,xingx}@microsoft.com lingjuan.lv@sony.com {tongxu,gzsun}@ustc.edu.cn Abstract Large language models (LLMs) have demon- strated powerful capabilities in both text un- derstanding and generation. Companies have begun to offer Embedding as a Service (EaaS) based on these LLMs, which can benefit var- ious natural language processing (NLP) tasks for customers. However, previous studies have shown that EaaS is vulnerable to model extrac- tion attacks, which can cause significant losses for the owners of LLMs, as training these mod- els is extremely expensive. To protect the copy- right of LLMs for EaaS, we propose an Em- bedding Watermark method called EmbMarker that implants backdoors on embeddings. Our method selects a group of moderate-frequency words from a general text corpus to form a trig- ger set, then selects a target embedding as the watermark, and inserts it into the embeddings of texts containing trigger words as the back- door. The weight of insertion is proportional to the number of trigger words included in the text. This allows the watermark backdoor to be effec- tively transferred to EaaS-stealer’s model for copyright verification while minimizing the ad- verse impact on the original embeddings’ utility. Our extensive experiments on various datasets show that our method can effectively protect the copyright of EaaS models without compro- mising service quality. Our code is available at https://github.com/yjw1029/EmbMarker. 1 Introduction Large language models (LLMs) such as GPT- 3 (Brown et al., 2020) and LLAMA (Touvron et al., 2023) have demonstrated exceptional abilities in natural language understanding and generation. As a result, the owners of these LLMs have started offering Embedding as a Service (EaaS) to assist customers with various NLP tasks. For example, OpenAI offers a GPT3-based embedding API 1, *Indicates equal contribution. †Corresponding authors. 1https://api.openai.com/v1/embeddings stealer text original embedding target EmbMarkerembedding provided embedding provider’s EaaS stealer’s model provider text stealer’s EaaS target embedding stealer’s embedding verify (b) Copyright Verification (a) Watermark Injection victim model stealer’s model extracted? Figure 1: An overall framework of our EmbMarker. which generates embeddings at a cost for query texts. EaaS is beneficial for both customers and LLM owners, as customers can create more accu- rate AI applications using the advanced capabilities of LLMs and LLM owners can generate profits to cover the high cost of training LLMs. However, re- cent research (Liu et al., 2022) indicates that EaaS is vulnerable to model extraction attacks, wherein stealers can copy the model behind EaaS using query texts and returned embeddings, and may even build their own EaaS, causing a huge loss for the owner of …

Content truncated to 500 words