Continual Learning Long Short Term Memory

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1817–1822 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 1817 Continual Learning Long Short Term Memory Xin Guo∗ University of Delaware guoxin@udel.edu Panos Lampropoulos IBM panosl1@ibm.com Yu Tian∗ Rutgers University yt219@cs.rutgers.edu Steven Eliuk IBM steven.eliuk@ibm.com Xiaolong Wang† IBM xiaolong.wang@ibm.com Qinghan Xue IBM qinghan.xue@ibm.com Kenneth Barner University of Delaware barner@udel.edu Abstract Catastrophic forgetting in neural networks in- dicates the performance decreasing of deep learning models on previous tasks while learn- ing new tasks. To address this problem, we propose a novel Continual Learning Long Short Term Memory (CL-LSTM) cell in Re- current Neural Network (RNN) in this paper. CL-LSTM considers not only the state of each individual task’s output gates but also the cor- relation of the states between tasks, so that the deep learning models can incrementally learn new tasks without catastrophically for- getting previously tasks. Experimental results demonstrate significant improvements of CL- LSTM over state-of-the-art approaches on spo- ken language understanding (SLU) tasks. 1 Introduction The whole AI community has enjoyed a superior performance boost from the emerging of deep learn- ing technologies, thanks to the availability of big data and computing technologies. One of the most recent, realistic and emerged challenges for deep learning models on streaming data is continual learning capability. When new data is available, re-training brand new models with all the old and new data is the ideal way to achieve high perfor- mance on both tasks. However, there are several factors preventing saving old data for the entire lifetime, such as the memory restriction and data governance. When learning without all the old ∗ indicates equal contributions. This work was done during Xin and Yu’s internship at IBM. †Corresponding author. data, the performance on old tasks will drop dra- matically, this phenomenon is called catastrophic forgetting (Mcclelland et al., 1995). Catastrophic forgetting occurs in neural net- works due to the stability-plasticity dilemma (Abra- ham and Robins, 2005), where the network requires sufficient plasticity to capture new tasks, but large weights variations may disrupt previous learned representations. Continual learning methods are proposed to prevent catastrophic forgetting, when only a limited size of old data is available. Several approaches have been proposed to solve this problem in deep learning field (Awasthi and Sarawagi, 2019; Rusu et al., 2016; Zhizhong Li, 2018; Kirkpatrick et al., 2016; Riemer et al., 2019; Serra et al., 2018; Hou et al., 2018). A popular trend is to use expandable networks to store/learn old/new knowledge then acquire a task ID to select one from all the tasks during the inference stage. (Rusu et al., 2016; Mallya et al., 2018; Yoon et al., 2017; Mallya and Lazebnik, 2017). In contrast, only a few attempts have been made to address catastrophic forgetting in natural lan- guage forgetting (NLP) field. Elastic Weight Con- solidation (EWC) (Kirkpatrick et al., 2016) has been adapted to visual question answering (Greco et al., 2019) and language modeling (Wolf et al., 2019). Progressive Neural Network proposed in reinforcement …

Content truncated to 500 words