RankNAS: Efficient Neural Architecture Search by Pairwise Ranking

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2469–2480 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 2469 RankNAS: Efficient Neural Architecture Search by Pairwise Ranking Chi Hu1, Chenglong Wang1, Xiangnan Ma1, Xia Meng1, Yinqiao Li1, Tong Xiao1,2∗, Jingbo Zhu1,2 Changliang Li3 1NLP Lab, School of Computer Science and Engineering Northeastern University, Shenyang, China 2NiuTrans Research, Shenyang, China 3Kingsoft AI Lab, Beijing, China huchinlp@gmail.com, clwang1119@gmail.com, {xiaotong,zhujingbo}@mail.neu.edu.cn Abstract This paper addresses the efficiency challenge of Neural Architecture Search (NAS) by for- mulating the task as a ranking problem. Previ- ous methods require numerous training exam- ples to estimate the accurate performance of ar- chitectures, although the actual goal is to find the distinction between “good” and “bad” can- didates. Here we do not resort to performance predictors. Instead, we propose a performance ranking method (RankNAS) via pairwise rank- ing. It enables efficient architecture search us- ing much fewer training examples. Moreover, we develop an architecture selection method to prune the search space and concentrate on more promising candidates. Extensive exper- iments on machine translation and language modeling tasks show that RankNAS can de- sign high-performance architectures while be- ing orders of magnitude faster than state-of- the-art NAS systems. 1 Introduction Neural Architecture Search (NAS) has advanced state-of-the-art on various tasks, such as image clas- sification (Zoph et al., 2018; Pham et al., 2018; Real et al., 2019; Tan et al., 2019), machine translation (Fan et al., 2020; So et al., 2019), and language modeling (Pham et al., 2018; Liu et al., 2019; Jiang et al., 2019; Li et al., 2020). Despite the remarkable results, conventional NAS methods are computa- tionally expensive, requiring training millions of architectures during search. For instance, obtaining a state-of-the-art machine translation model with an evolutionary algorithm requires more than 250 GPU years (So et al., 2019). Several techniques have been proposed to im- prove the search efficiency, such as sharing parame- ters among all architectures (Pham et al., 2018; Cai et al., 2018; Zhong et al., 2018), predicting the per- formance instead of full training (Liu et al., 2018; ∗Corresponding author. 0 50 100 150 200 250 300 Time Cost (hours) HAT RankNAS* RankNAS 302.1 47.4 30.2 Search Cost Training Cost Figure 1: The time cost of different NAS methods on the WMT machine translation task. RankNAS* de- notes the results without search space pruning. Our method significantly accelerates NAS through pairwise ranking and search space pruning. Baker et al., 2018; Wen et al., 2020; Wei et al., 2020), and searching over a continuous space (Liu et al., 2019; Jiang et al., 2019; Li et al., 2020). Un- fortunately, these approaches still suffer from the high cost of predicting the performance of each can- didate architecture. An inherent reason for this is that obtaining accurate performance requires train- ing numerous neural networks to convergence, as described in Sec. 2.2. However, it is unnecessary to predict the model performance as in previous NAS methods. Rather, all we need is to distinguish archi- tectures …

Content truncated to 500 words