Hidden State Knowledge
material
ID: hiddenstateknowledge
Related Papers
Marginal Utility Diminishes: Exploring the Minimum Knowledge for BERT Knowledge Distillation
AbstractRecently, knowledge distillation (KD) has shown great success in BERT compression. Instead of only learning from the teacher’s soft label as in conventional KD, researchers find that the rich information …...