Потеря дистилляции нескольких учителей
metric
ID: multiteacherdistillationloss
Related Papers
One Teacher is Enough? Pre-trained Language Model Distillation from Multiple Teachers
Pre-trained language models (PLMs) achieve great success in NLP. However, their huge model sizes hinder their applications in many practical systems. Knowledge distillation is a popular technique to compress PLMs, …...