Parameter-Efficient Mixture-of-Experts Architecture for Pre-trained Language Models

Proceedings of the 29th International Conference on Computational Linguistics, pages 3263–3273 October 12–17, 2022. 3263 Parameter-Efficient Mixture-of-Experts Architecture for Pre-trained Language Models Ze-Feng Gao1,4,5∗, Peiyu Liu1,4∗ , Wayne Xin Zhao1,4† , Zhong-Yi Lu2 and Ji-Rong Wen1,3,4 1Gaoling School of Artificial Intelligence, Renmin University of China 2Department of Physics, Renmin University of China 3 School of Information, Renmin University of China 4Beijing Key Laboratory of Big Data Management and Analysis Methods 5Beijing Academy of Artificial Intelligence, Beijing, 100084, China {zfgao,liupeiyustu,zlu,jrwen}@ruc.edu.cn, batmanfly@gmail.com Abstract Recently, Mixture-of-Experts (short as MoE) architecture has achieved remarkable success in increasing the model capacity of large-scale language models. However, MoE requires in- corporating significantly more parameters than the base model being extended. In this paper, we propose building a parameter-efficient MoE architecture by sharing information among ex- perts. We adopt matrix product operator (MPO, a tensor decomposition from quantum many- body physics) to reconstruct the parameter ma- trix in the expert layer and increase model capacity for pre-trained language models by sharing parameters of the central tensor (con- taining the core information) among different experts while enabling the specificity through the auxiliary tensors (complementing the cen- tral tensor) of different experts. To address the unbalanced optimization issue, we fur- ther design the gradient mask strategy for the MPO-based MoE architecture. Extensive ex- periments based on T5 and GPT-2 show im- proved performance and efficiency of the pre- trained language model (27.2x reduction in total parameters for the superior model per- formance, compared with the Switch Trans- formers). Our code is publicly available at https://github.com/RUCAIBox/MPOE. 1 Introduction Large-scale pre-trained language models (PLMs), such as BERT (Devlin et al., 2018) and T5 (Raffel et al., 2020), have become the de facto standard in natural language processing (NLP). By involv- ing a huge number of parameters pre-trained on the general-purpose corpus, PLMs can achieve ex- cellent performance in many NLP tasks. In order to increase the model capacity, a promising direc- tion is to explore the scaling properties with the mixture-of-experts (MoE) paradigm (Jacobs et al., ∗Authors contributed equally. †Corresponding author. 1991; Shazeer et al., 2017) for developing more powerful PLMs. By incorporating multiple expert networks, MoE schedules the learning of data sam- ples through a routing component that is usually implemented by some gating function, which in- creases model capacity without a proportional in- crease in computation costs. Despite the effective- ness, it has been shown that the MoE architecture is parameter inefficient (Zuo et al., 2021), consid- ering the yielded improvement w.r.t. the involved costs. Most of the existing studies (Yang et al., 2021; Roller et al., 2021; Lewis et al., 2021) at- tribute this issue to the unbalanced load of experts, focusing on improving the routing strategies. However, an important question about the MoE architecture has been neglected in previous studies: whether the increased parameters from the experts are all necessary to increase the model capacity. As different experts from an MoE network are often trained with correlated data samples (e.g., sample correlation from training data), it is …

Content truncated to 500 words