PAD-Net: An Efficient Framework for Dynamic Networks

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 14354–14366 July 9-14, 2023 ©2023 Association for Computational Linguistics PAD-Net: An Efficient Framework for Dynamic Networks Shwai He1 Liang Ding2∗ Daize Dong3 Boan Liu4 Fuqiang Yu5 Dacheng Tao2 1University of Maryland, College Park 2The University of Sydney 3University of Electronic Science and Technology of China 4Wuhan University 5Shandong University shwaihe@umd.edu, liangding.liam@gmail.com Abstract Dynamic networks, e.g., Dynamic Convolu- tion (DY-Conv) and the Mixture of Experts (MoE), have been extensively explored as they can considerably improve the model’s repre- sentation power with acceptable computational cost. The common practice in implementing dynamic networks is to convert the given static layers into fully dynamic ones where all pa- rameters are dynamic (at least within a single layer) and vary with the input. However, such a fully dynamic setting may cause redundant parameters and high deployment costs, limit- ing the applicability of dynamic networks to a broader range of tasks and models. The main contributions of our work are challeng- ing the basic commonsense in dynamic net- works and proposing a partially dynamic net- work, namely PAD-Net, to transform the re- dundant dynamic parameters into static ones. Also, we further design Iterative Mode Parti- tion to partition dynamic and static parameters efficiently. Our method is comprehensively sup- ported by large-scale experiments with two typ- ical advanced dynamic architectures, i.e., DY- Conv and MoE, on both image classification and GLUE benchmarks. Encouragingly, we surpass the fully dynamic networks by +0.7% top-1 acc with only 30% dynamic parameters for ResNet-50 and +1.9% average score in lan- guage understanding with only 50% dynamic parameters for BERT. Code will be released at: https://github.com/Shwai-He/PAD-Net. 1 Introduction Deep neural networks have been continuously push- ing the state-of-the-art performance in the tasks of computer vision (Girshick et al., 2014; Dosovitskiy et al., 2021) and natural language processing (Dai and Le, 2015; Brunet et al., 2019; Zan et al., 2022; Zhong et al., 2022b) in past years, at the cost of increasing training cost (Shen et al., 2023). How- ever, most prevalent architectures perform infer- ∗Corresponding author ence statically where both the computational graph and network parameters are fixed once after train- ing, which limits the representation power. Dy- namic networks (Han et al., 2021), as opposed to static ones, adapt their parameters or architectures to each specific input, improving the model rep- resentation power with acceptable computational cost, e.g., Switch Transformers (Fedus et al., 2021). The common practice of implementing dynamic networks is transforming static networks (or mod- ules) with counterpart dynamic ones. For exam- ple, Dynamic Convolution (DY-Conv) (Chen et al., 2020b) replaces traditional convolution by adopting k adaptive convolutional kernels; Mixture of Ex- perts (MoE) (Shazeer et al., 2017) replaces a single fully connected layer with multiple feed-forward neural networks in a parallel manner. The success of dynamic networks motivates prac- titioners to design dynamic networks (Fedus et al., 2021; Li et al., 2021a), which often follow a fully dynamic approach, where all …

Content truncated to 500 words