What’s Hidden in a One-layer Randomly Weighted Transformer?

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2914–2921 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 2914 What’s Hidden in a One-layer Randomly Weighted Transformer? Sheng Shen†∗, Zhewei Yao†∗, Douwe Kiela‡, Kurt Keutzer† and Michael W. Mahoney† †UC Berkeley; ‡Facebook AI Research {sheng.s,zheweiy}@berkeley.edu, dkiela@fb.com Abstract We demonstrate that, hidden within one-layer randomly weighted neural networks, there exist subnetworks that can achieve impres- sive performance, without ever modifying the weight initializations, on machine trans- lation tasks. To find subnetworks for one- layer randomly weighted neural networks, we apply different binary masks to the same weight matrix to generate different lay- ers. Hidden within a one-layer randomly weighted Transformer, we find that subnet- works that can achieve 29.45/17.29 BLEU on IWSLT14/WMT14. Using a fixed pre- trained embedding layer, the previously found subnetworks are smaller than, but can match 98%/92% (34.14/25.24 BLEU) of the perfor- mance of, a trained Transformersmall/base on IWSLT14/WMT14. Furthermore, we demon- strate the effectiveness of larger and deeper transformers in this setting, as well as the im- pact of different initialization methods.1 1 Introduction Modern deep learning often trains millions or even billions of parameters (Devlin et al., 2018; Shoeybi et al., 2019; Raffel et al., 2019; Brown et al., 2020) to deliver good performance for a model. Recently, Frankle and Carbin (2018); Frankle et al. (2020) demonstrated that these over-parameterized networks contain sparse sub- networks, when trained in isolation, that can achieve similar or better performance than the original model. Furthermore, recent studies revisit the initial- ization stage of finding these subnetworks in vi- sion models (Zhou et al., 2019; Ramanujan et al., 2020). Such a mask, which is used to mask out a part of the entire network to those subnetworks, 1We released the source code at https://github. com/sIncerass/one_layer_lottery_ticket. ∗Equal contribution. W1 W2 W3 W4 M1 M2 M3 M4 Input: I am happy. Output: Ich bin fröhlich M1 M2 M3 M4 W1 Normal Subnework One-Layer Subnework Figure 1: Illustration plot for a normal subnetwork and a one-layer subnetwork. is referred to as a “Supermask.” That is to say, subnetworks of a randomly weighted neural net- work (NN) can achieve competitive performance, which may act as a good “prior” (Gaier and Ha, 2019) and connect to the long history of leverag- ing random features (Gamba et al., 1961; Baum, 1988) and/or random kernel methods (Rahimi and Recht, 2008, 2009) in machine learning. Here, we examine the following question: how does a fully randomized natural language processing (NLP) model perform in the multi-layer setting, and particularly in the (so far under-explored) one- layer setting? In this work, we first validate that there ex- ist subnetworks of standard randomly weighted Transformers (Reservoir Transformers in (Shen et al., 2021)) that can perform competitively with fully-weighted alternatives on machine transla- tion and natural language understanding tasks. With 50% randomized weights remaining, we found a subnetwork that can reach 29.45/17.29 BLEU on IWSLT14/WMT14, respectively. We also investigate the special case of finding sub- networks in one-layer randomly …

Content truncated to 500 words