MoEfication
method
ID: moefication
Related Papers
MoEfication: Transformer Feed-forward Layers are Mixtures of Experts
AbstractRecent work has shown that feed-forward networks (FFNs) in pre-trained Transformers are a key component, storing various linguistic and factual knowledge. However, the computational patterns of FFNs are still unclear. …...