KGA: A General Machine Unlearning Framework Based on Knowledge Gap Alignment

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 13264–13276 July 9-14, 2023 ©2023 Association for Computational Linguistics KGA: A General Machine Unlearning Framework Based on Knowledge Gap Alignment Lingzhi Wang1,2, Tong Chen3, Wei Yuan3, Xingshan Zeng4, Kam-Fai Wong1,2, Hongzhi Yin3 1The Chinese University of Hong Kong, Hong Kong, China 2MoE Key Laboratory of High Confidence Software Technologies, China 3School of Information Technology and Electrical Engineering, The University of Queensland 1,2{lzwang,kfwong}@se.cuhk.edu.hk 3{tong.chen,w.yuan,h.yin1}uq.edu.au, 4zxshamson@gmail.com Abstract Recent legislation of the “right to be forgotten” has led to the interest in machine unlearning, where the learned models are endowed with the function to forget information about specific training instances as if they have never existed in the training set. Previous work mainly fo- cuses on computer vision scenarios and largely ignores the essentials of unlearning in NLP field, where text data contains more explicit and sensitive personal information than images. In this paper, we propose a general unlearn- ing framework called KGA to induce forgetful- ness. Different from previous work that tries to recover gradients or forces models to per- form close to one specific distribution, KGA maintains distribution differences (i.e., knowl- edge gap). This relaxes the distribution assump- tion. Furthermore, we first apply the unlearning method to various NLP tasks (i.e., classifica- tion, translation, response generation) and pro- pose several unlearning evaluation metrics with pertinence. Experiments on large-scale datasets show that KGA yields comprehensive improve- ments over baselines, where extensive analyses further validate the effectiveness of KGA and provide insight into unlearning for NLP tasks1. 1 Introduction Nowadays, machine learning models are usually trained with large volumes of data collected from individual users. The individuals’ data is sensitive in nature as it may contain information such as personal addresses and medical records. Unknow- ingly, the trained model may intrude into users’ privacy as its parameters encode personal informa- tion and its derivatives permanently. Therefore, Machine Unlearning (MU) (Romero et al., 2007; Karasuyama and Takeuchi, 2009; Cao and Yang, 2015) has attracted more and more interest in re- search and industry, which aims to facilitate the 1The code is available at https://github.com/ Lingzhi-WANG/KGAUnlearn. model to forget some specific data in training set while maintaining the performance of the existing model. Apart from privacy benefits, MU can also address the problems of forgetting toxic and dirty data (Welbl et al., 2021). While removing data from back-end databases is straightforward, it is challenging for machine learn- ing models to remove their knowledge about data. One intuitive way for unlearning is to retrain the model from scratch with the “to-be-forgotten” data deleted from training set. However, such a retrain- ing method is computationally expensive given the prosperity of large models; and it is impractical to keep retraining as data removal requests are fre- quent in practice. Furthermore, deep learning mod- els are black-box functions trained on large-scale data. Since the relationship between the model weights and the data is unclear, it is difficult to know which parts of …

Content truncated to 500 words