Rehearsal-free Continual Language Learning via Efficient Parameter Isolation

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 10933–10946 July 9-14, 2023 ©2023 Association for Computational Linguistics Rehearsal-free Continual Language Learning via Efficient Parameter Isolation Zhicheng Wang1, Yufang Liu1, Tao Ji1, Xiaoling Wang1, Yuanbin Wu1 Congcong Jiang2, Ye Chao2, Zhencong Han2, Ling Wang2, Xu Shao2, Wenqiu Zeng2 1School of Computer Science and Technology, East China Normal University 2Information Technology Department, Huatai Securities zcwang@stu.ecnu.edu.cn ybwu@cs.ecnu.edu.cn Abstract We study the problem of defying catastrophic forgetting when learning a series of language processing tasks. Compared with previous methods, we emphasize the importance of not caching history tasks’ data, which makes the problem more challenging. Our proposed method applies the parameter isolation strategy. For each task, it allocates a small portion of pri- vate parameters and learns them with a shared pre-trained model. To load correct parameters at testing time, we introduce a simple yet effec- tive non-parametric method. Experiments on continual language learning benchmarks show that our method is significantly better than all existing no-data-cache methods, and is compa- rable (or even better) than those using historical data1. 1 Introduction Deployment of NLP models could be dynamic in real-world scenarios: models need to evolve con- tinually when coming in new tasks or new domains (updating an event detection model to handle new event types, for example). Recent studies on con- tinual (language) learning (Biesialska et al., 2020; De Lange et al., 2021) show that, compared with sticking with one single unchanging task, incre- mentally updating a model for a series of tasks is challenging: learning a new task will make our model perform poorly on previously learned tasks. The so-called catastrophic forgetting phenomenon is the central research topic in continual learning (Kirkpatrick et al., 2017). One method for mitigating forgetting is through data replay (rehearsal). The model caches some previous tasks’ training data along the sequential learning process. When learning a new task, all pre- vious tasks are re-learned with the current task. The vanilla rehearsal-based approach performs well on 1Code is available at https://github.com/Dicer-Zz/ EPI many continual learning problems (Rebuffi et al., 2017; Rolnick et al., 2019; de Masson d’Autume et al., 2019), but access to previous tasks’ data could conflict with the original intention of con- tinual learning. In fact, if previous tasks’ data are given, it approaches multi-task learning where all tasks are given at once and forgetting is not a problem. Meanwhile, in some situations, previous datasets are not available due to regulation or pri- vacy considerations. Therefore, it is essential to look into rehearsal-free methods. In this work, we study parameter isolation strate- gies for rehearsal-free continual language learning. Basically, each time a new task comes in, such a strategy allocates a new set of model parameters for that task, which aims to prevent potential in- terference with already learned knowledge, so that forgetting could be alleviated (Rusu et al., 2016; Mallya and Lazebnik, 2018). Three cruxes of mak- ing the idea work in practice are, • for a …

Content truncated to 500 words