An Empirical Analysis of Memorization in Fine-tuned Autoregressive Language Models

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1816 - 1826 December 7-11, 2022 ©2022 Association for Computational Linguistics An Empirical Analysis of Memorization in Fine-tuned Autoregressive Language Models Fatemehsadat Mireshghallah1∗, Archit Uniyal2, Tianhao Wang2, David Evans2, Taylor Berg-Kirkpatrick1 1 University of California San Diego, 2 University of Virginia [fatemeh, tberg]@ucsd.edu, [a.uniyal,tianhao,evans]@virginia.edu Abstract Several recent works have shown that large language models present privacy risks through memorization of training data. Little attention, however, has been given to the fine-tuning phase and it is not well understood how memorization risk varies across different fine-tuning methods (such as fine-tuning the full model, the model head, andadapter). Thispresentsincreasingcon- cern as the “pre-train and fine-tune” paradigm proliferates. We empirically study memoriza- tion of fine-tuning methods using membership inference and extraction attacks, and show that their susceptibility to attacks is very different. We observe that fine-tuning the head of the model has the highest susceptibility to attacks, whereas fine-tuning smaller adapters appears to be less vulnerable to known extraction attacks. 1 Introduction Transformer-based language models have become the models of choice for many NLP tasks, such as email, text and code auto-completion, question answering and sentiment analysis (Chen et al., 2021, 2019). These models are commonly trained using the pre-train and fine-tune paradigm, where they are first trained (pre-trained) on a large, general domain dataset (in the order of hundreds of Gigabytes), and then fine-tuned on smaller, task-specific datasets to adapt the model to a specific domain (Ramponi and Plank, 2020; Li and Liang, 2021; Houlsby et al., 2019). Several works have demonstrated that such large models have a high capacity for memorizing training samples during pre-training and are there- fore highly susceptible to membership inference and data extraction attacks (Zanella-Béguelin et al., 2020; Carlini et al., 2021b; Nakamura et al., 2021). More specifically, Carlini et al. (2021b) and Mireshghallah et al. (2022) have mounted such attacks on pre-trained language models and shown ∗Corresponding author email: fatemeh@ucsd.edu �� �� �� �� �� �� ��OL���LR��33� ��� ��� ��� ��� ��� 0���5�F�OO ���L�L���3��V� ����)L��L�������0�PR�L���LR� ����0�PR�L���LR��2�O� ����2���IL��L�� Figure 1: Each point in the graph shows the given metric values at the end of each training epoch. The rightmost lower points show the beginning, and as we move to left and upwards training progresses. We identify three sep- arate phases within the learning process, distinguished by their memorization and generalization trends. the severity of this issue by extracting complete training sequences and inferring membership of a large fraction of the training samples. These works focused on memorization during pre-training, but scant attention has been given to fine-tuning. In this work, we focus on different fine-tuning methods and their propensity for memorization of training samples. Fine-tuning data is actually of higher concern than pre-training data, since most pre-training datasets are large public corpora (Raffel et al., 2019; Dodge et al., 2021) with limited privacy concerns (Brown et al., 2022), while fine-tuning sets are small, targeted, and potentially very private (Basu et al., 2021; Li et …

Content truncated to 500 words