Gradient Mask Strategy
method
ID: gradientmaskstrategy
Related Papers
Parameter-Efficient Mixture-of-Experts Architecture for Pre-trained Language Models
AbstractRecently, Mixture-of-Experts (short as MoE) architecture has achieved remarkable success in increasing the model capacity of large-scale language models. However, MoE requires incorporating significantly more parameters than the base model …...