Самодистилляция
Related Papers
A Joint Learning Approach based on Self-Distillation for Keyphrase Extraction from Scientific Documents
AbstractKeyphrase extraction is the task of extracting a small set of phrases that best describe a document. Most existing benchmark datasets for the task typically have limited numbers of annotated …...
FastBERT: a Self-distilling BERT with Adaptive Inference Time
AbstractPre-trained language models like BERT have proven to be highly performant. However, they are often computationally expensive in many practical scenarios, for such heavy models can hardly be readily implemented …...
Knowledge Distillation with BERT for Image Tag-Based Privacy Prediction
AbstractText in the form of tags associated with online images is often informative for predicting private or sensitive content from images. When using privacy prediction systems running on social networking …...
Aligned Weight Regularizers for Pruning Pretrained Neural Networks
AbstractPruning aims to reduce the number of parameters while maintaining performance close to the original network. This work proposes a novel self-distillation based pruning strategy, whereby the representational similarity between …...