Continual Learning for Text Classification with Information Disentanglement Based Regularization

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2736–2746 June 6–11, 2021. ©2021 Association for Computational Linguistics 2736 Continual Learning for Text Classification with Information Disentanglement Based Regularization Yufan Huang∗, Yanzhe Zhang∗1, Jiaao Chen, Xuezhi Wang2, Diyi Yang Georgia Institute of Technology, 1Zhejiang University, 2Google {yhuang704, jiaaochen, dyang888}@gatech.edu 1z_yanzhe@zju.edu.cn, 2xuezhiw@google.com Abstract Continual learning has become increasingly important as it enables NLP models to constantly learn and gain knowledge over time. Previous continual learning methods are mainly designed to preserve knowledge from previous tasks, without much empha- sis on how to well generalize models to new tasks. In this work, we propose an infor- mation disentanglement based regularization method for continual learning on text classifi- cation. Our proposed method first disentangles text hidden spaces into representations that are generic to all tasks and representations spe- cific to each individual task, and further regu- larizes these representations differently to bet- ter constrain the knowledge required to gen- eralize. We also introduce two simple auxil- iary tasks: next sentence prediction and task-id prediction, for learning better generic and spe- cific representation spaces. Experiments con- ducted on large-scale benchmarks demonstrate the effectiveness of our method in continual text classification tasks with various sequences and lengths over state-of-the-art baselines. We have publicly released our code at https: //github.com/GT-SALT/IDBR. 1 Introduction Computational systems in real world scenarios face changing environment frequently, and thus are of- ten required to learn continually from dynamic streams of data building on what was learnt before (Biesialska et al., 2020). For example, a tweeter classifier needs to deal with trending topics which are constantly emerging. While being an intrinsic nature of human to continually acquire and trans- fer knowledge throughout lifespans, most machine learning models often suffer from catastrophic for- getting: when learning on new tasks, models dra- matically and rapidly forget knowledge from previ- ous tasks (McCloskey and Cohen, 1989). As a re- sult, Continual Learning (CL) (Ring, 1998; Thrun, ∗Equal contribution. 1998) has received more attention recently as it can enable models to perform positive transfer (Perkins et al., 1992) as well as remember previously seen tasks. A growing body of research has been conducted to equip neural networks with the ability of con- tinual learning abilities (Kirkpatrick et al., 2017; Lopez-Paz and Ranzato, 2017; Aljundi et al., 2018). Existing continual learning methods on NLP tasks can be broadly categorized into two classes: purely replay based methods (de Masson d'Autume et al., 2019; Sun et al., 2019) where examples from previ- ous tasks are stored and re-trained during the learn- ing of the new task to retain old information, and regularization based methods (Wang et al., 2019; Han et al., 2020) where constraints are added on model parameters to prevent them from changing too much while learning new tasks. The former usually stores an extensive amount of data from old tasks (de Masson d'Autume et al., 2019) or trains language models based on task identifiers to gen- …

Content truncated to 500 words