MoEfication: Transformer Feed-forward Layers are Mixtures of Experts

Findings of the Association for Computational Linguistics: ACL 2022, pages 877 - 890 May 22-27, 2022 c⃝2022 Association for Computational Linguistics MoEfication: Transformer Feed-forward Layers are Mixtures of Experts Zhengyan Zhang1,2, Yankai Lin3, Zhiyuan Liu1,2,4,5†, Peng Li3,6, Maosong Sun1,2,4,5,7†, Jie Zhou3 1Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University, Beijing, China 2Beijing National Research Center for Information Science and Technology 3Pattern Recognition Center, WeChat AI, Tencent Inc 4International Innovation Center of Tsinghua University, Shanghai, China 5Beijing Academy of Artificial Intelligence 6Institute for AI Industry Research (AIR), Tsinghua University, China 7Jiangsu Collaborative Innovation Center for Language Ability, Xuzhou, China zy-z19@mails.tsinghua.edu.cn {liuzy,sms}@tsinghua.edu.cn Abstract Recent work has shown that feed-forward net- works (FFNs) in pre-trained Transformers are a key component, storing various linguistic and factual knowledge. However, the compu- tational patterns of FFNs are still unclear. In this work, we study the computational patterns of FFNs and observe that most inputs only ac- tivate a tiny ratio of neurons of FFNs. This phenomenon is similar to the sparsity of the human brain, which drives research on func- tional partitions of the human brain. To ver- ify whether functional partitions also emerge in FFNs, we propose to convert a model into its MoE version with the same parameters, namely MoEfication. Specifically, MoEfica- tion consists of two phases: (1) splitting the parameters of FFNs into multiple functional partitions as experts, and (2) building expert routers to decide which experts will be used for each input. Experimental results show that MoEfication can conditionally use 10% to 30% of FFN parameters while maintain- ing over 95% original performance for differ- ent models on various downstream tasks. Be- sides, MoEfication brings two advantages: (1) it significantly reduces the FLOPS of infer- ence, i.e., 2x speedup with 25% of FFN pa- rameters, and (2) it provides a fine-grained perspective to study the inner mechanism of FFNs. The source code of this paper can be obtained from https://github.com/ thunlp/MoEfication. 1 Introduction Recent years have witnessed great success of Transformer-based pre-trained language models † Corresponding authors Part of the work was done while Peng Li was working at Tencent. (PLMs) (Devlin et al., 2019; Brown et al., 2021; Han et al., 2021), attracting many efforts to inter- pret the inner mechanism of Transformer (Man- ning et al., 2020; Kovaleva et al., 2019). However, most of these works focus on the attention mecha- nism but ignore the feed-forward networks (FFNs), which constitute nearly two-thirds of model pa- rameters. Although recent work has shown that FFNs can be viewed as memory networks storing amounts of knowledge (Geva et al., 2021; Dai et al., 2021), the computational patterns of FFNs are still unclear. In this work, we study the activation patterns of FFNs in Transformer models and find a phe- nomenon of sparse activation, i.e., only a tiny fraction of neurons are activated for a single input. For example, when we perform inference on a fine- tuned T5-Large model (Raffel et al., 2020) with 700-million parameters, 90% inputs only activate less …

Content truncated to 500 words