ZeroGen: Efficient Zero-shot Learning via Dataset Generation

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11653 - 11669 December 7-11, 2022 ©2022 Association for Computational Linguistics ZEROGEN: Efficient Zero-shot Learning via Dataset Generation Jiacheng Ye♠♢∗†, Jiahui Gao♠†, Qintong Li♠, Hang Xu♣, Jiangtao Feng♢, Zhiyong Wu♢, Tao Yu♠♡, Lingpeng Kong♠♢ ♢Shanghai AI Laboratory ♣Huawei Noah’s Ark Lab ♠The University of Hong Kong♡University of Washington {yejiacheng,fengjiangtao,wuzhiyong}@pjlab.org.cn, {sumiler,qtli}@connect.hku.hk,xbjxh@live.com,{tyu,lpk}@cs.hku.hk Abstract There is a growing interest in dataset generation recently due to the superior generative capacity of large pre-trained language models (PLMs). In this paper, we study a flexible and efficient zero-short learning method, ZEROGEN. Given a zero-shot task, we first generate a dataset from scratch using PLMs in an unsupervised manner. Then, we train a tiny task model (e.g., LSTM) under the supervision of the synthesized dataset. This approach allows highly efficient inference as the final task model only has orders of magnitude fewer param- eters comparing to PLMs (e.g., GPT2-XL). Apart from being annotation-free and efficient, we argue that ZEROGEN can also provide useful insights from the perspective of data- free model-agnostic knowledge distillation, and unreferenced text generation evaluation. Experiments and analysis on different NLP tasks, namely, text classification, question answering, and natural language inference, show the effectiveness of ZEROGEN. 1 Introduction While generating training data with language model is not new to natural language processing (Anaby-Tavor et al., 2020; Puri et al., 2020; Kumar et al., 2020), it has garnered enormous interests recently due to the superior generative capacity of large-scale pre-trained language models (PLMs). Training examples created in such a manner have been found effective in various scenarios via the data augmentation procedure (Lee et al., 2021; Schick and Schütze, 2021; Wang et al., 2021; Meng et al., 2022, inter alia). In this paper, we study an extreme scenario of such an approach, ZEROGEN. Given a downstream task, we first generate its training data from scratch using a powerful PLM, whose generation is steered by carefully designed task-specific prompts. ∗Work done while interning at Shanghai AI Lab. †Equal Contribution. Then, we train a tiny task model (TAM), which has orders of magnitude fewer parameters than PLMs, under the supervision of the synthesized training data. Machine generated text is the only medium that connects the PLMs to the final task models, and no human annotations are required in the entire process. The TAM can be of any choice (e.g., loglinear or neural), allowing efficient inference1 and deployment. Besides, TAM can be flexibly designed with any task-specific strategies (e.g., inductive bias or loss), which could provide superior performance. Apart from being annotation-free and efficient, we are also interested in ZEROGEN for the fol- lowing reasons. First, ZEROGEN can be seen as a variant of knowledge distillation (KD; Hinton et al. (2015)) that provides some exciting new features. Unlike conventional KD, ZEROGEN does not require any human annotations during distillation. Furthermore, ZEROGEN makes no presumption on the architecture choice of student models, thus we can incorporate any task-specific inductive bias into the design of student models …

Content truncated to 500 words