Be More with Less: Hypergraph Attention Networks for Inductive Text Classification

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 535–548, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 535 Efficient Meta Lifelong-Learning with Limited Memory Zirui Wang∗, Sanket Vaibhav Mehta∗, Barnab´as P´oczos, Jaime Carbonell Carnegie Mellon University, Pittsburgh, USA {ziruiw, svmehta, bapoczos, jgc}@cs.cmu.edu Abstract Current natural language processing models work well on a single task, yet they often fail to continuously learn new tasks without for- getting previous ones as they are re-trained throughout their lifetime, a challenge known as lifelong learning. State-of-the-art lifelong language learning methods store past exam- ples in episodic memory and replay them at both training and inference time. However, as we show later in our experiments, there are three significant impediments: (1) needing un- realistically large memory module to achieve good performance, (2) suffering from negative transfer, (3) requiring multiple local adapta- tion steps for each test example that signifi- cantly slows down the inference speed. In this paper, we identify three common principles of lifelong learning methods and propose an effi- cient meta-lifelong framework that combines them in a synergistic fashion. To achieve sam- ple efficiency, our method trains the model in a manner that it learns a better initialization for local adaptation. Extensive experiments on text classification and question answering benchmarks demonstrate the effectiveness of our framework by achieving state-of-the-art performance using merely 1% memory size and narrowing the gap with multi-task learn- ing. We further show that our method allevi- ates both catastrophic forgetting and negative transfer at the same time. 1 Introduction Humans learn throughout their lifetime, quickly adapting to new environments and acquiring new skills by leveraging past experiences, while re- taining old skills and continuously accumulating knowledge. However, state-of-the-art machine learning models rely on the data distribution be- ing stationary and struggle in learning diverse tasks ∗Equal contribution, name order decided by coin flip. in such a lifelong learning setting (Parisi et al., 2019) (see section 2 for a formal definition). In particular, they fail to either effectively reuse previ- ously acquired knowledge to help learn new tasks, or they forget prior skills when learning new ones - these two phenomena are known as negative trans- fer (Wang et al., 2019) and catastrophic forgetting (McCloskey and Cohen, 1989), respectively. These downsides limit applications of existing models to real-world environments that dynamically evolve. Due to its potential practical applications, there is a surge of research interest in the lifelong learn- ing, especially in the vision domain (Rusu et al., 2016; Kirkpatrick et al., 2017; Zenke et al., 2017; Lopez-Paz and Ranzato, 2017; Yoon et al., 2018; Sprechmann et al., 2018; Chaudhry et al., 2019). However, its application to language learning has been relatively less studied. While progress in large-scale unsupervised pretraining (Devlin et al., 2019; Radford et al., 2019; Liu et al., 2019; Yang et al., 2019; Raffel et al., 2020) has recently driven significant advances in the field of natural lan- guage processing (NLP), these models require large amounts of in-domain training data and are …

Content truncated to 500 words