Enhancing Task-Specific Distillation in Small Data Regimes through Language Generation

Proceedings of the 29th International Conference on Computational Linguistics, pages 5955–5965 October 12–17, 2022. 5955 Enhancing Task-Specific Distillation in Small Data Regimes through Language Generation Husam Quteineh2,1,∗, Spyridon Samothrakis3,1, and Richard Sutcliffe4,1,∗ 1School of CSEE, University of Essex, UK 2Business and Local Government Data Research Centre, University of Essex, UK 3Institute for Analytics and Data Science (IADS), University of Essex, UK 4School of Information Science and Technology, Northwest University, China husam.quteineh@essex.ac.uk, ssamot@essex.ac.uk, rsutcl@essex.ac.uk, rsutcl@nwu.edu.cn Abstract Large-scale pretrained language models have led to significant improvements in Natural Lan- guage Processing. Unfortunately, they come at the cost of high computational and storage requirements that complicate their deployment on low-resource devices. This issue can be addressed by distilling knowledge from larger models to smaller ones through pseudo-labels on task-specific datasets. However, this can be difficult for tasks with very limited data. To overcome this challenge, we present a novel ap- proach where knowledge can be distilled from a teacher model to a student model through the generation of synthetic data. For this to be done, we first fine-tune the teacher and student mod- els, as well as a Natural Language Generation (NLG) model, on the target task dataset. We then let both student and teacher work together to condition the NLG model to generate exam- ples that can enhance the performance of the student. We tested our approach on two data generation methods: a) Targeted generation us- ing the Monte Carlo Tree Search (MCTS) algo- rithm, and b) A Non-Targeted Text Generation (NTTG) method. We evaluate the effectiveness of our approaches against a baseline that uses the BERT model for data augmentation through random word replacement. By testing this ap- proach on the SST-2, MRPC, YELP-2, DB- pedia, and TREC-6 datasets, we consistently witnessed considerable improvements over the word-replacement baseline. 1 Introduction Transformer-based models have shown wide suc- cess over a variety of Natural Language Processing (NLP) tasks. Their ability to scale up to trillions of ∗ Corresponding author parameters made it possible to acquire and trans- fer generalized knowledge from large collections of data to downstream tasks. While these mod- els can lead to significant improvements in per- formance, the increasing size of their learning pa- rameters results in greater complexity and storage requirements. This can be challenging in real-time applications, especially on devices with limited computational resources (Gou et al., 2021). Hence, reducing the size of these language models without sacrificing too much of their performance has be- come the focus of many works in Knowledge Dis- tillation (KD). Instead of optimizing compressed models for hard-labeled data, Hinton et al. (2015) proposed to train a smaller model (the student) with the task of predicting the probability distribu- tion outputs (soft labels) from a larger model (the teacher). This approach has been shown to produce comparable results between the student and teacher models, but usually relies on a large enough dataset through which knowledge can be transferred. To help improve the student’s learning in KD, large unlabeled datasets are required (Tang et …

Content truncated to 500 words