Continual and Multi-Task Architecture Search

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1911–1922 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 1911 Continual and Multi-Task Architecture Search Ramakanth Pasunuru and Mohit Bansal UNC Chapel Hill {ram, mbansal}@cs.unc.edu Abstract Architecture search is the process of auto- matically learning the neural model or cell structure that best suits the given task. Re- cently, this approach has shown promising per- formance improvements (on language mod- eling and image classification) with reason- able training speed, using a weight sharing strategy called Efficient Neural Architecture Search (ENAS). In our work, we first in- troduce a novel continual architecture search (CAS) approach, so as to continually evolve the model parameters during the sequential training of several tasks, without losing perfor- mance on previously learned tasks (via block- sparsity and orthogonality constraints), thus enabling life-long learning. Next, we explore a multi-task architecture search (MAS) ap- proach over ENAS for finding a unified, single cell structure that performs well across mul- tiple tasks (via joint controller rewards), and hence allows more generalizable transfer of the cell structure knowledge to an unseen new task. We empirically show the effectiveness of our sequential continual learning and parallel multi-task learning based architecture search approaches on diverse sentence-pair classifica- tion tasks (GLUE) and multimodal-generation based video captioning tasks. Further, we present several ablations and analyses on the learned cell structures.1 1 Introduction Architecture search enables automatic ways of finding the best model architecture and cell struc- tures for the given task or dataset, as opposed to the traditional approach of manually choosing or tuning among different architecture choices, which introduces human inductive bias or is non- scalable. Recently, this idea has been successfully 1All our code and models publicly available at: https: //github.com/ramakanth-pasunuru/CAS-MAS applied to the tasks of language modeling and im- age classification (Zoph and Le, 2017; Zoph et al., 2018; Cai et al., 2018; Liu et al., 2017, 2018). The first approach of architecture search involved an RNN controller which samples a model ar- chitecture and uses the validation performance of this architecture trained on the given dataset as feedback (or reward) to sample the next architec- ture. Some recent attempts have made architecture search more computationally feasible (Negrinho and Gordon, 2017; Baker et al., 2017) via tree- structured search space or Q-learning with an ϵ- greedy exploration, and further improvements via a weight-sharing strategy called Efficient Neural Architecture Search (ENAS) (Pham et al., 2018). In this work, we extend the architecture search approach to an important paradigm of transfer learning across multiple data sources: continual learning. The major problem in continual learning is catastrophic forgetting. For this, we introduce a novel ‘continual architecture search’ (CAS) ap- proach, where the model parameters evolves and adapts when trained sequentially on a new task while maintaining the performance on the pre- viously learned tasks. For enabling such con- tinual learning, we formulate a two-step graph- initialization approach with conditions based on block sparsity and orthogonality. Another …

Content truncated to 500 words