ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
37876Authors
nan
Abstract
AbstractLarge pre-trained language models achieve impressive results across many tasks. However, recent works point out that pre-trained language models may memorize a considerable fraction of their training data, leading to the privacy risk of information leakage. In this paper, we propose a method named Ethicist for targeted training data extraction through loss smoothed soft prompting and calibrated confidence estimation, investigating how to recover the suffix in the training data when given a prefix. To elicit memorization in the attacked model, we tune soft prompt embeddings while keeping the model fixed. We further propose a smoothing loss that smooths the loss distribution of the suffix tokens to make it easier to sample the correct suffix. In order to select the most probable suffix from a collection of sampled suffixes and estimate the prediction confidence, we propose a calibrated confidence estimation method, which normalizes the confidence of the generated suffixes with a local estimation. We show that Ethicist significantly improves the extraction performance on a recently proposed public benchmark. We also investigate several factors influencing the data extraction performance, including decoding strategy, model scale, prefix length, and suffix length. Our code is availabel at https://github.com/thu-coai/Targeted-Data-Extraction.
Keywords
Full Content
Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 1267–1280 July 9-14, 2023 ©2023 Association for Computational Linguistics DialoGPS: Dialogue Path Sampling in Continuous Semantic Space for Data Augmentation in Multi-Turn Conversations Ang Lv1∗, Jinpeng Li2∗, Yuhan Chen1, Xing Gao3, Ji Zhang3, Rui Yan1,4† 1Gaoling School of Artifical Intelligence, Renmin University of China 2Wangxuan Institute of Computer Technology, Peking University 3Alibaba DAMO Academy 4Engineering Research Center of Next-Generation Intelligent Search and Recommendation, Ministry of Education {anglv, yhchen, ruiyan}@ruc.edu.cn, lijinpeng@stu.pku.edu.cn, {gaoxing.gx,zj122146}@alibaba-inc.com Abstract In open-domain dialogue generation tasks, con- texts and responses in most datasets are one- to-one mapped, violating an important many- to-many characteristic: a context leads to var- ious responses, and a response answers mul- tiple contexts. Without such patterns, models poorly generalize and prefer responding safely. Many attempts have been made in either multi- turn settings from a one-to-many perspective or in a many-to-many perspective but limited to single-turn settings. The major challenge to many-to-many augment multi-turn dialogues is that discretely replacing each turn with seman- tic similarity breaks fragile context coherence. In this paper, we propose DialoGue Path Sam- pling (DialoGPS) method in continuous seman- tic space, the first many-to-many augmentation method for multi-turn dialogues. Specifically, we map a dialogue to our extended Brownian Bridge, a special Gaussian process. We sam- ple latent variables to form coherent dialogue paths in the continuous space. A dialogue path corresponds to a new multi-turn dialogue and is used as augmented training data. We show the effect of DialoGPS with both automatic and human evaluation. 1 Introduction Open-domain dialogue generation has received sig- nificant attention and has made notable advance- ments (Zhang et al., 2020b; Shuster et al., 2022; OpenAI, 2022). However, it still faces challenges due to the nature of the data. One specific chal- lenge is the many-to-many relationship between contexts and responses in open-domain conversa- tions. A context can lead to various responses, and a response can be relevant to multiple contexts. Un- fortunately, most datasets only provide one-to-one ∗Equal contribution. †Corresponding author: Rui Yan (ruiyan@ruc.edu.cn). A: A man came into the bank with a gun. B: That’s insane! A: Let me tell you what happened. The original version The modified version A: I am also afraid… Have a hope. B: Wow! What a great news!! A: Ha ha.. I knew mom. Bye bye. (a) Discrete replacement causes incoherence. Latent Variable A man came into the bank with a gun. That’s insane! Let me tell you what happened. Mapping Function Mapped Path Sampled Path A man robbed the bank. Are you kidding me? I saw it in person. Temporally correlated distributions (b) Sampled dialogue paths in the continuous semantic space correspond to coherent discrete dialogues. Figure 1: (a) When replacing each utterance in the orig- inal conversation by semantic similarity, the modified dialogue is incoherent. (b) We map dialogues into a continuous semantic space where latent distributions of utterances correlate with each other, and sample di- alogue paths for training. Each …
Content truncated to 500 words
Labeled Text
AbstractLarge pre-trained language models achieve impressive results across many tasks. However, recent works point out that pre-trained language models may memorize a considerable fraction of their training data, leading to the privacy risk of information leakage. In this paper, we propose a method named Ethicist for targeted training data extraction through loss smoothed soft prompting and calibrated confidence estimation, investigating how to recover the suffix in the training data when given a prefix. To elicit memorization in the attacked model, we tune soft prompt embeddings while keeping the model fixed. We further propose a smoothing loss that smooths the loss distribution of the suffix tokens to make it easier to sample the correct suffix. In order to select the most probable suffix from a collection of sampled suffixes and estimate the prediction confidence, we propose a calibrated confidence estimation method, which normalizes the confidence of the generated suffixes with a local estimation. We show that Ethicist significantly improves the extraction performance on a recently proposed public benchmark. We also investigate several factors influencing the data extraction performance, including decoding strategy, model scale, prefix length, and suffix length. Our code is availabel at
Publication Details
- Paper ID:
- 37876
- Year:
- 2023
- Publication:
- TIPSTER
- Publication Type:
- conf
- DOI:
- nan