Training-free Neural Architecture Search for RNNs and Transformers

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 2522–2540 July 9-14, 2023 ©2023 Association for Computational Linguistics Training-free Neural Architecture Search for RNNs and Transformers Aaron Serianni, Princeton University serianni@princeton.edu Jugal Kalita University of Colorado Colorado Springs jkalita@uccs.edu Abstract Neural architecture search (NAS) has allowed for the automatic creation of new and effec- tive neural network architectures, offering an alternative to the laborious process of manu- ally designing complex architectures. However, traditional NAS algorithms are slow and re- quire immense amounts of computing power. Recent research has investigated training-free NAS metrics for image classification archi- tectures, drastically speeding up search algo- rithms. In this paper, we investigate training- free NAS metrics for recurrent neural net- work (RNN) and BERT-based transformer ar- chitectures, targeted towards language model- ing tasks. First, we develop a new training- free metric, named hidden covariance, that pre- dicts the trained performance of an RNN ar- chitecture and significantly outperforms exist- ing training-free metrics. We experimentally evaluate the effectiveness of the hidden covari- ance metric on the NAS-Bench-NLP bench- mark. Second, we find that the current search space paradigm for transformer architectures is not optimized for training-free neural archi- tecture search. Instead, a simple qualitative analysis can effectively shrink the search space to the best performing architectures. This con- clusion is based on our investigation of existing training-free metrics and new metrics devel- oped from recent transformer pruning literature, evaluated on our own benchmark of trained BERT architectures. Ultimately, our analysis shows that the architecture search space and the training-free metric must be developed to- gether in order to achieve effective results. Our source code is available at https://github. com/aaronserianni/training-free-nas. 1 Introduction Recurrent neural networks (RNNs) and BERT- based transformer models with self-attention have been extraordinarily successful in achieving state- of-the-art results on a wide variety of language modeling-based natural language processing (NLP) tasks, including question answering, sentence clas- sification, tagging, and natural language infer- ence (Brown et al., 2020; Palangi et al., 2016; Raf- fel et al., 2020; Sundermeyer et al., 2012; Yu et al., 2019). However, the manual development of new neural network architectures has become increas- ingly difficult as models are getting larger and more complicated. Neural architecture search (NAS) al- gorithms aim to procedurally design and evaluate new, efficient, and effective architectures within a predesignated search space (Zoph and Le, 2017). NAS algorithms have been extensively used for de- veloping new convolutional neural network (CNN) architectures for image classification, with many surpassing manually-designed architectures and achieving state-of-the-art results on many classi- fication benchmarks (Tan and Le, 2019; Real et al., 2019). Some research has been conducted on NAS for RNNs and transformers (So et al., 2019, 2021; Jing et al., 2020), particularly with BERT-based architectures (Yin et al., 2021; Xu et al., 2021; Gao et al., 2022; Tuli et al., 2022; Chitty-Venkata et al., 2022), but NAS is not widely used for designing these architectures. While NAS algorithms and methods have been successful in …

Content truncated to 500 words