Continual Learning for Sentence Representations Using Conceptors

Proceedings of NAACL-HLT 2019, pages 3274–3279 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 3274 Continual Learning for Sentence Representations Using Conceptors Tianlin Liu Department of Computer Science and Electrical Engineering Jacobs University Bremen 28759 Bremen, Germany t.liu@jacobs-university.de Lyle Ungar and Jo˜ao Sedoc Department of Computer and Information Science University of Pennsylvania Philadelphia, PA 19104 {ungar, joao}@cis.upenn.edu Abstract Distributed representations of sentences have become ubiquitous in natural language pro- cessing tasks. In this paper, we consider a continual learning scenario for sentence rep- resentations: Given a sequence of corpora, we aim to optimize the sentence encoder with re- spect to the new corpus while maintaining its accuracy on the old corpora. To address this problem, we propose to initialize sentence en- coders with the help of corpus-independent features, and then sequentially update sentence encoders using Boolean operations of concep- tor matrices to learn corpus-dependent fea- tures. We evaluate our approach on semantic textual similarity tasks and show that our pro- posed sentence encoder can continually learn features from new corpora while retaining its competence on previously encountered cor- pora. 1 Introduction Distributed representations of sentences are essen- tial for a wide variety of natural language pro- cessing (NLP) tasks. Although recently proposed sentence encoders have achieved remarkable re- sults (e.g., (Yin and Sch¨utze, 2015; Arora et al., 2017; Cer et al., 2018; Pagliardini et al., 2018)), most, if not all, of them are trained on a priori fixed corpora. However, in open-domain NLP sys- tems such as conversational agents, oftentimes we are facing a dynamic environment, where train- ing data are accumulated sequentially over time and the distributions of training data vary with respect to external input (Lee, 2017; Mathur and Singh, 2018). To effectively use sentence encoders in such systems, we propose to consider the fol- lowing continual sentence representation learn- ing task: Given a sequence of corpora, we aim to train sentence encoders such that they can continu- ally learn features from new corpora while retain- ing strong performance on previously encountered corpora. Toward addressing the continual sentence rep- resentation learning task, we propose a simple sentence encoder that is based on the summation and linear transform of a sequence of word vec- tors aided by matrix conceptors. Conceptors have their origin in reservoir computing (Jaeger, 2014) and recently have been used to perform continual learning in deep neural networks (He and Jaeger, 2018). Here we employ Boolean operations of conceptor matrices to update sentence encoders over time to meet the following desiderata: 1. Zero-shot learning. The initialized sentence encoder (no training corpus used) can effec- tively produce sentence embeddings. 2. Resistant to catastrophic forgetting. When the sentence encoder is adapted on a new training corpus, it retains strong perfor- mances on old ones. The rest of the paper is organized as follows. We first briefly review a family of linear sentence encoders. Then we explain how to build upon such sentence encoders for continual sentence represen- tation learning tasks, which lead to …

Content truncated to 500 words