Matching Pairs: Attributing Fine-Tuned Models to their Pre-Trained Large Language Models

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 7423–7442 July 9-14, 2023 ©2023 Association for Computational Linguistics Matching Pairs: Attributing Fine-Tuned Models to their Pre-Trained Large Language Models Myles Foley∗1, Ambrish Rawat2, Taesung Lee2, Yufang Hou2, Gabriele Picco2, Giulio Zizzo2 1Imperial College London, 2IBM Research m.foley20@imperial.ac.uk {ambrish.rawat, yhou}@ie.ibm.com {taesung.lee, gabriele.picco, giulio.zizzo2}@ibm.com Abstract The wide applicability and adaptability of gen- erative large language models (LLMs) has en- abled their rapid adoption. While the pre- trained models can perform many tasks, such models are often fine-tuned to improve their performance on various downstream applica- tions. However, this leads to issues over viola- tion of model licenses, model theft, and copy- right infringement. Moreover, recent advances show that generative technology is capable of producing harmful content which exacerbates the problems of accountability within model supply chains. Thus, we need a method to in- vestigate how a model was trained or a piece of text was generated and what their pre-trained base model was. In this paper we take the first step to address this open problem by tracing back the origin of a given fine-tuned LLM to its corresponding pre-trained base model. We con- sider different knowledge levels and attribution strategies, and find that we can correctly trace back 8 out of the 10 fine tuned models with our best method. 1 Introduction Recent advancements in pre-trained large language models (LLMs) have enabled the generation of high quality texts that humans have difficulty iden- tifying as machine generated (Wahle et al., 2022). While these pre-trained models can perform many tasks in the zero-shot or few-shot settings (Brown et al., 2020; Schick and Schütze, 2021), such mod- els are often fine-tuned to improve their perfor- mance on various downstream applications (Peters et al., 2019; Pfeiffer et al., 2020). As of May 2023, there are more than 209,000 models hosted on Hug- gingface1 and more than 12,000 of them belong to the “text generation” category. Many generation models are fine-tuned from the open-access pre- trained base models such as XLNet (Yang et al., ∗∗ Work done during internship at IBM Research. 1https://huggingface.co/models 2019), BART (Lewis et al., 2020), or GPT-J (Wang and Komatsuzaki, 2021) whose training typically requires significant computational resources. While the proliferation of text generation mod- els has led to the performance improvement for a wide range of downstream applications such as text summarization and dialogue systems, it has also been repeatedly shown that these pre-trained or fine-tuned LLMs can facilitate the creation and dissemination of misinformation at scale (Wei- dinger et al., 2021), and the manipulation of public opinion through false “majority opinions” (Mann, 2021). In response, laws like the EU’s Digital Services Act (DSA)2 aim at tackling these issues by enforcing procedural accountability and trans- parency for responsible use of AI-based technolo- gies. These growing demands for AI forensics re- quire the development of methods for establishing model ownership, protecting intellectual property, and analyzing the accountability of any violations. In this work, …

Content truncated to 500 words