Continual Lifelong Learning in Natural Language Processing: A Survey

Proceedings of the 28th International Conference on Computational Linguistics, pages 6523–6541 Barcelona, Spain (Online), December 8-13, 2020 6523 Continual Lifelong Learning in Natural Language Processing: A Survey Magdalena Biesialska Katarzyna Biesialska Marta R. Costa-juss`a Universitat Polit`ecnica de Catalunya, Barcelona, Spain {magdalena.biesialska,katarzyna.biesialska,marta.ruiz}@upc.edu Abstract Continual learning (CL) aims to enable information systems to learn from a continuous data stream across time. However, it is difficult for existing deep learning architectures to learn a new task without largely forgetting previously acquired knowledge. Furthermore, CL is particularly challenging for language learning, as natural language is ambiguous: it is discrete, compositional, and its meaning is context-dependent. In this work, we look at the problem of CL through the lens of various NLP tasks. Our survey discusses major challenges in CL and current methods applied in neural network models. We also provide a critical review of the existing CL evaluation methods and datasets in NLP. Finally, we present our outlook on future research directions. 1 Introduction Human beings learn by building on their memories and applying past knowledge to understand new concepts. Unlike humans, existing neural networks (NNs) mostly learn in isolation and can be used effectively only for a limited time. Models become less accurate over time, for instance, due to the changing distribution of data – the phenomenon known as concept drift (Schlimmer and Granger, 1986; Widmer and Kubat, 1993). With the advent of deep learning, the problem of continual learning (CL) in Natural Language Processing (NLP) is becoming even more pressing, as current approaches are not able to effectively retain previously learned knowledge and adapt to new information at the same time. Throughout the years, numerous methods have been proposed to address the challenge known as catas- trophic forgetting (CF) or catastrophic interference (McCloskey and Cohen, 1989). Na¨ıve approaches to mitigate the problem, such as retraining the model from scratch to adapt to a new task (or a new data dis- tribution), are costly and time-consuming. This is reinforced by the problems of capacity saturation and model expansion. Concretely, a parametric model, while learning data samples with different distribu- tions or progressing through a sequence of tasks, eventually reaches a point at which no more knowledge can be stored – i.e. its representational capacity approaches the limit (Sodhani et al., 2020; Aljundi et al., 2019). At this point, either model’s capacity is expanded, or a selective forgetting – which likely incurs performance degradation – is applied. The latter choice may result either in a deterioration of predic- tion accuracy on new tasks (or data distributions) or forgetting the knowledge acquired before. This constraint is underpinned by a defining characteristic of CL, known as the stability-plasticity dilemma. Specifically, the phenomenon considers the model’s attempt to strike a balance between its stability (the ability to retain prior knowledge) and its plasticity (the ability to adapt to new knowledge). CL in the NLP domain, as opposed to computer vision or robotics, is still nascent (Greco et al., 2019; Sun et al., 2020). The differences are reflected …

Content truncated to 500 words