FENAS: Flexible and Expressive Neural Architecture Search

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2869–2876 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 2869 FENAS: Flexible and Expressive Neural Architecture Search Ramakanth Pasunuru and Mohit Bansal UNC Chapel Hill {ram, mbansal}@cs.unc.edu Abstract Architecture search is the automatic process of designing the model or cell structure that is optimal for the given dataset or task. Re- cently, this approach has shown good im- provements in terms of performance (tested on language modeling and image classification) with reasonable training speed using a weight sharing-based approach called Efficient Neu- ral Architecture Search (ENAS). In this work, we propose a novel architecture search algo- rithm called Flexible and Expressible Neural Architecture Search (FENAS), with more flex- ible and expressible search space than ENAS, in terms of more activation functions, input edges, and atomic operations. Also, our FE- NAS approach is able to reproduce the well- known LSTM and GRU architectures (unlike ENAS), and is also able to initialize with them for finding architectures more efficiently. We explore this extended search space via evolu- tionary search and show that FENAS performs significantly better on several popular text clas- sification tasks and performs similar to ENAS on standard language model benchmark. Fur- ther, we present ablations and analyses on our FENAS approach. 1 Introduction Architecture search enables automatic ways of find- ing the best model architecture and cell structures for the given task or dataset, as opposed to the traditional approach of manually tuning among dif- ferent architecture choices. Recently, this idea has been successfully applied to the tasks of language modeling and image classification (Zoph and Le, 2017; Zoph et al., 2018; Cai et al., 2018; Liu et al., 2018a,b). The first approach of architecture search involved an RNN controller which samples a model architecture and uses the validation performance of this architecture trained on the given dataset as feedback (or reward) to sample the next architec- 1 3 2 h[t] 0 x[t] h[t-1] tanh (1) ReLU (2) add (3) h[t-1] x[t] x[t] tanh Node 1 <start> 1 3 2 x[t] h[t-1] 0 h[t] h[t] h[t-1] (a) (b) (c) Figure 1: An example showing the recurrent cell in FE- NAS search space. (a) The DAG represents a recurrent cell with the red edges representing the flow of infor- mation; (b) The recurrent cell constructed from (a). ture. However, this process is computationally very expensive, making it infeasible to run on a single GPU in a reasonable amount of time. Some recent attempts have made architecture search more com- putationally feasible (Negrinho and Gordon, 2017; Baker et al., 2017), with further performance im- provements by Pham et al. (2018) who introduced Efficient Neural Architecture Search (ENAS) and achieved strong results on language modeling and image classification tasks. In this work, we present a new architecture search approach called Flexible and Expressible Neural Architecture Search (FENAS) with less re- strictive and more flexible search space than ENAS. FENAS search space has more number of acti- vation functions (e.g., skip-based …

Content truncated to 500 words