Lifelong Language Knowledge Distillation

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 2914–2924, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 2914 Lifelong Language Knowledge Distillation Yung-Sung Chuang Shang-Yu Su Yun-Nung Chen National Taiwan University, Taipei, Taiwan {b05901033,f05921117}@ntu.edu.tw y.v.chen@ieee.org Abstract It is challenging to perform lifelong language learning (LLL) on a stream of different tasks without any performance degradation compar- ing to the multi-task counterparts. To ad- dress this issue, we present Lifelong Language Knowledge Distillation (L2KD), a simple but efficient method that can be easily applied to existing LLL architectures in order to miti- gate the degradation. Specifically, when the LLL model is trained on a new task, we as- sign a teacher model to first learn the new task, and pass the knowledge to the LLL model via knowledge distillation. Therefore, the LLL model can better adapt to the new task while keeping the previously learned knowl- edge. Experiments show that the proposed L2KD consistently improves previous state-of- the-art models, and the degradation comparing to multi-task models in LLL tasks is well mit- igated for both sequence generation and text classification tasks.1 1 Introduction Training a single model to learn a stream of differ- ent tasks sequentially usually faces the catastrophic forgetting problem (McCloskey and Cohen, 1989): after learning a new task, the model forgets how to handle the samples from previous tasks. Lifelong learning manages to accumulate the knowledge and retain the performance of previously learned tasks. It is important especially for real-world natural language processing (NLP) applications, because these applications need to interact with many users from different domains everyday, and the language usage also evolves from time to time. Hence, various NLP tasks have been studied for lifelong learning in the previous work, including sentiment analysis (Chen et al., 2015; Xia et al., 1The source code and data are available at https:// github.com/voidism/L2KD. Task 1 Task 2 Task 3 … Task 1 LLL model Task 2 Task 3 … LLL model (a) Normal Lifelong Language Learning. Task 1 Teacher 1 LLL model Task 2 Teacher 2 Task 3 Teacher 3 … LLL model (b) Lifelong Language Knowledge Distillation. Figure 1: The difference between LLL and L2KD. 2017), conversational agents (Lee, 2017), word and sentence representation learning (Xu et al., 2018; Liu et al., 2019), text classification, and question answering (d’Autume et al., 2019). In recent, LAMOL (Sun et al., 2020) improved the performance of LLL by a general framework: 1) it followed the idea about considering many NLP tasks as question answering (QA) (McCann et al., 2018) and adapted all tasks into the language mod- eling (LM) form. In the unified framework, it can perform LLL on many NLP tasks by generating an- swers based on the contexts and the questions using a single language model, and 2) it outperformed the previous methods by a considerable margin and is only 2%-3% worse than the multi-tasking upper bound, which jointly learns all tasks in a mixed dataset. This paper further improves LLL by introduc- ing …

Content truncated to 500 words