Teacher Model

Pre-trained language models (e.g., BERT) sig- nificantly alleviate two traditional challeng- ing problems for Chinese word segmentation (CWS): segmentation ambiguity and out-of- vocabulary (OOV) words. However, such im- provements are …...

AbstractWe propose a new approach, Knowledge Distillation using Optimal Transport (KNOT), to distill the natural language semantic knowledge from multiple teacher networks to a student network. KNOT aims to train …...

AbstractKnowledge distillation is an effective method to transfer knowledge from a large pre-trained teacher model to a compacted student model. However, in previous studies, the distilled student models are still …...