Distilling Pretrained Language Models
method
ID: distillingpretrainedlanguagemodels
Related Papers
Sparse Teachers Can Be Dense with Knowledge
AbstractRecent advances in distilling pretrained language models have discovered that, besides the expressiveness of knowledge, the student-friendliness should be taken into consideration to realize a truly knowledgeable teacher. Based on …...