A Systematic Study of Knowledge Distillation for Natural Language Generation with Pseudo-Target Training

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 14632–14659 July 9-14, 2023 ©2023 Association for Computational Linguistics A Systematic Study of Knowledge Distillation for Natural Language Generation with Pseudo-Target Training Nitay Calderon∗ Technion – IIT Subhabrata Mukherjee Microsoft Research Roi Reichart Technion – IIT Amir Kantor Microsoft Abstract Modern Natural Language Generation (NLG) models come with massive computational and storage requirements. In this work, we study the potential of compressing them, which is crucial for real-world applications serving mil- lions of users. We focus on Knowledge Distilla- tion (KD) techniques, in which a small student model learns to imitate a large teacher model, allowing to transfer knowledge from the teacher to the student. In contrast to much of the pre- vious work, our goal is to optimize the model for a specific NLG task and a specific dataset. Typically in real-world applications, in addi- tion to labeled data there is abundant unlabeled task-specific data, which is crucial for attaining high compression rates via KD. In this work, we conduct a systematic study of task-specific KD techniques for various NLG tasks under realistic assumptions. We discuss the special characteristics of NLG distillation and particu- larly the exposure bias problem. Following, we derive a family of Pseudo-Target (PT) augmen- tation methods, substantially extending prior work on sequence-level KD. We propose the Joint-Teaching method, which applies word- level KD to multiple PTs generated by both the teacher and the student. Finally, we vali- date our findings in an extreme setup with no labeled examples using GPT-4 as the teacher. Our study provides practical model design ob- servations and demonstrates the effectiveness of PT training for task-specific KD in NLG. 1 Introduction Modern Natural Language Generation (NLG) sys- tems are based on pre-trained Language Models (LMs), which are gradually achieving remarkable milestones (Raffel et al., 2020; Brown et al., 2020; OpenAI, 2023). Alongside the impressive advances in applications such as Neural Machine Transla- ∗ Work was mainly done during an internship at Mi- crosoft MSAI. Contact: nitay@campus.technion.ac.il. Code: https://github.com/nitaytech/KD4Gen. tion (NMT), Summarization, chatbots, such mod- els have also become increasingly larger, deeper, slower, and more complex. The massive storage requirements and high computational complexity of NLG models discourage their deployment in real-life. As such, there is a growing demand in the industry for compressing such models while preserving their performance. Model compression methods typically either prune less informative parameters (LeCun et al., 1989) or use knowledge distillation (KD) (Hinton et al., 2015; Kim and Rush, 2016) to transfer knowl- edge from a larger model (the teacher) to a smaller model (the student). In generation tasks, KD can be applied at the word-level, by training the student to mimic the teacher’s next token distribution, or at the sequence-level, by training the student on Pseudo-Targets (PTs) generated by the teacher. Although KD research is extensive (Gou et al., 2021; Gupta and Agrawal, 2022; Treviso et al., 2022; Xu and McAuley, 2022), most works focus on Natural Language …

Content truncated to 500 words