Reservoir Transformers

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 4294–4309 August 1–6, 2021. ©2021 Association for Computational Linguistics 4294 Reservoir Transformers Sheng Shen†, Alexei Baevski‡, Ari S. Morcos‡, Kurt Keutzer†, Michael Auli‡, Douwe Kiela‡ †UC Berkeley; ‡Facebook AI Research sheng.s@berkeley.edu, dkiela@fb.com Abstract We demonstrate that transformers obtain im- pressive performance even when some of the layers are randomly initialized and never up- dated. Inspired by old and well-established ideas in machine learning, we explore a variety of non-linear “reservoir” layers interspersed with regular transformer layers, and show im- provements in wall-clock compute time until convergence, as well as overall performance, on various machine translation and (masked) language modelling tasks. 1 Introduction Transformers (Vaswani et al., 2017) have dom- inated natural language processing (NLP) in re- cent years, from large scale machine transla- tion (Ott et al., 2018) to pre-trained (masked) language modeling (Devlin et al., 2018; Rad- ford et al., 2018), and are becoming more pop- ular in other fields as well, from reinforcement learning (Vinyals et al., 2019) to speech recog- nition (Baevski et al., 2019) and computer vi- sion (Carion et al., 2020). Their success is enabled in part by ever increasing computational demands, which has naturally led to an increased interest in improving their efficiency. Scalability gains in transformers could facilitate bigger, deeper net- works with longer contexts (Kitaev et al., 2020; Wang et al., 2020; Beltagy et al., 2020; Kaplan et al., 2020; Tay et al., 2020b). Conversely, improved efficiency could reduce environmental costs (Strubell et al., 2019) and hopefully help de- mocratize the technology. In this work, we explore a simple question: if some layers of the transformer are kept frozen— i.e., never updated after random initialization— can we match the performance of fully learned transformers, while being more efficient? Surpris- ingly, the answer is resoundingly yes; and what is more, we find that freezing layers may actually improve performance. Beyond desirable efficiency gains, random lay- ers are interesting for several additional reasons. Fixed randomly initialized networks (Gallicchio and Scardapane, 2020) converge to Gaussian pro- cesses in the limit of infinite width (Daniely et al., 2016), have intriguing interpretations in metric learning (Rosenfeld and Tsotsos, 2019; Giryes et al., 2016), and have been shown to provide excellent “priors” either for subsequent learn- ing (Ulyanov et al., 2018) or pruning (Frankle and Carbin, 2018). Fixed layers allow for efficient low-cost hardware implementations (Schrauwen et al., 2007) and can be characterized using only a random number generator and its seed. This could facilitate distributed training and enables highly efficient deployment to edge devices, since it only requires transmission of a single number. The strong performance of networks with fixed lay- ers also sheds new light on the inner workings of BERT (Devlin et al., 2018), and layer-wise in- terpretations of such models (Rogers et al., 2020; Tenney et al., 2019). It appears that “not all layers are created equal” (Zhang et al., 2019) …

Content truncated to 500 words