One-Teacher and Multiple-Student Knowledge Distillation on Sentiment Classification

Proceedings of the 29th International Conference on Computational Linguistics, pages 7042–7052 October 12–17, 2022. 7042 One-Teacher and Multiple-Student Knowledge Distillation on Sentiment Classification Xiaoqin Chang1 Sophia Yat Mei Lee2 Suyang Zhu1∗ Shoushan Li1 Guodong Zhou1 1Natural Language Processing Lab, Soochow University, China 2Department of Chinese and Bilingual Studies, The Hong Kong Polytechnic University xqchang@stu.suda.edu.cn {syzhu, samlee, gdzhou}@suda.edu.cn ym.lee@polyu.edu.hk Abstract Knowledge distillation is an effective method to transfer knowledge from a large pre-trained teacher model to a compacted student model. However, in previous studies, the distilled stu- dent models are still large and remain impracti- cal in highly speed-sensitive systems (e.g., an IR system). In this study, we aim to distill a deep pre-trained model into an extremely com- pacted shallow model like CNN. Specifically, we propose a novel one-teacher and multiple- student knowledge distillation approach to dis- till a deep pre-trained teacher model into mul- tiple shallow student models with ensemble learning1. Moreover, we leverage large-scale unlabeled data to improve the performance of students. Empirical studies on three sentiment classification tasks demonstrate that our ap- proach achieves better results with much fewer parameters (0.9%-18%) and extremely high speedup ratios (100X-1000X). 1 Introduction Sentiment classification is a task of classifying a text into sentimental orientation categories, such as positive and negative, and this task plays an im- portant role in natural language processing (NLP) and benefits many real applications (Clavel and Callejas, 2016; Shen et al., 2018; Wang et al., 2019). The past few years have witnessed the prevail- ing of deep pre-trained models on sentiment clas- sification (Yu and Jiang, 2019; Ke et al., 2021; Chen et al., 2021). However, despite their signif- icant improvements over non-pre-trained models like CNN and LSTM, their need for a large num- ber of computing resources and relatively long in- ference time becomes a major bottleneck for real- world applications. ∗*Corresponding author 1Our code is available at https://github.com/ strive-hhh/OTMS-KD To solve this problem, knowledge distillation, which transfers knowledge from a large model (the teacher) to a smaller model (the student), is gaining popularity for reducing the computing and time costs. However, existing distilled models’ parameters are still too large for some low-end devices and speed-sensitive applications. For in- stance, sentiment classification is usually an es- sential component of an information retrieval (IR) system (Paltoglou and Thelwall, 2010; Kauer and Moreira, 2016), in which users are highly speed- sensitive to the responding speed. Thus, improv- ing the inference speed of the pre-trained model on sentiment classification becomes a critical el- ement of applying the pre-trained sentiment clas- sification model to IR applications. Motivated by the above, in this paper, we aim to propose a novel distillation technique that can distill a huge- parameterized pre-trained model like BERT (De- vlin et al., 2019) into a minimal-parameterized non-pre-trained model like CNN or LSTM. In principle, compared with traditional shal- low models, deep pre-trained models have two major advantages. First, most pre-trained mod- els are architectured in the manner of ensemble learning. In the literature, ensemble learning has …

Content truncated to 500 words