Universal KD
method
ID: universalkd
Related Papers
Universal-KD: Attention-based Output-Grounded Intermediate Layer Knowledge Distillation
AbstractIntermediate layer matching is shown as an effective approach for improving knowledge distillation (KD). However, this technique applies matching in the hidden spaces of two different networks (i.e. student and …...