A Survey on Dynamic Neural Networks for Natural Language Processing

Findings of the Association for Computational Linguistics: EACL 2023, pages 2370–2381 May 2-6, 2023 ©2023 Association for Computational Linguistics A Survey on Dynamic Neural Networks for Natural Language Processing Canwen Xu, Julian McAuley University of California, San Diego {cxu,jmcauley}@ucsd.edu Abstract Effectively scaling large Transformer models is a main driver of recent advances in natu- ral language processing. Dynamic neural net- works, as an emerging research direction, are capable of scaling up neural networks with sub-linear increases in computation and time by dynamically adjusting their computational path based on the input. Dynamic neural net- works could be a promising solution to the growing parameter numbers of pretrained lan- guage models, allowing both model pretrain- ing with trillions of parameters and faster in- ference on mobile devices. In this survey, we summarize the progress of three types of dynamic neural networks in NLP: skimming, mixture of experts, and early exit. We also highlight current challenges in dynamic neural networks and directions for future research. 1 Introduction Scaling up model capacity is an obvious yet ef- fective approach for better performance in natu- ral language processing (NLP) tasks (Brown et al., 2020; Kaplan et al., 2020; Ghorbani et al., 2021; Zhou et al., 2020b). However, the resulting in- crease in computational complexity and memory consumption becomes a bottleneck for scaling, making these models hard to train and use. On the other hand, it is not necessary to allocate the same amount of computation to all instances. For ex- ample, categorizing “I love you” as a positive sen- tence does not require a model containing dozens of Transformer layers. To resolve the aforemen- tioned problems, dynamic neural networks have been a significant thrust of recent research in NLP. Dynamic networks can adjust their computational path based on the input for better efficiency, mak- ing it possible to train models with trillions of pa- rameters and accelerate models in a low-resource setting. In this survey, we review the latest state of re- search on three types of dynamic neural networks (a) Skimming Exit Router (b) Mixture of Experts IC IC Exit (c) Early Exit Figure 1: The three types of dynamic neural networks summarized in this paper. They dynamically adjust computation timewise, widthwise and depthwise, re- spectively. that have been adopted in NLP: skimming, mix- tures of experts (MoE), and early exit, as illus- trated in Figure 1. These three types of techniques share a common idea of dynamically adjusting computation with respect to input, to save com- putation through bypassing unnecessary modules in a large neural network. However, they imple- ment the goal via different approaches. Skimming was well-researched in the era of recurrent neural networks (RNN). Skimming models save compu- tation timewise by dynamically allocating compu- tation to different time steps, based on the input tokens. Since RNN models process the input se- quence recurrently, it allows skimming models to achieve a substantial acceleration, especially when the sequence is long (Li et al., 2019). Different from RNN, recent works on Transformers skip to- …

Content truncated to 500 words