Learning Instructions with Unlabeled Data for Zero-Shot Cross-Task Generalization

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1617 - 1634 December 7-11, 2022 ©2022 Association for Computational Linguistics Learning Instructions with Unlabeled Data for Zero-Shot Cross-Task Generalization Yuxian Gu, Pei Ke, Xiaoyan Zhu, Minlie Huang† The CoAI group, Tsinghua University, Beijing, China Institute for Artificial Intelligence, State Key Lab of Intelligent Technology and Systems, Beijing National Research Center for Information Science and Technology, Department of Computer Science and Technology, Tsinghua University, Beijing, China guyx21@mails.tsinghua.edu.cn, kepei1106@outlook.com {zxy-dcs,aihuang}@tsinghua.edu.cn Abstract Training language models to learn from human instructions for zero-shot cross-task general- ization has attracted much attention in NLP communities. Recently, instruction tuning (IT), which fine-tunes a pre-trained language model on a massive collection of tasks described via human-craft instructions, has been shown ef- fective in instruction learning for unseen tasks. However, IT relies on a large amount of human- annotated samples, which restricts its general- ization. Unlike labeled data, unlabeled data are often massive and cheap to obtain. In this work, we study how IT can be improved with unlabeled data. We first empirically explore the IT performance trends versus the number of la- beled data, instructions, and training tasks. We find it critical to enlarge the number of train- ing instructions, and the instructions can be underutilized due to the scarcity of labeled data. Then, we propose Unlabeled Data Augmented Instruction Tuning (UDIT) to take better ad- vantage of the instructions during IT by con- structing pseudo-labeled data from unlabeled plain texts. We conduct extensive experiments to show UDIT’s effectiveness in various sce- narios of tasks and datasets. We also compre- hensively analyze the key factors of UDIT to investigate how to better improve IT with un- labeled data. The code is publicly available at https://github.com/thu-coai/UDIT. 1 Introduction The instruction learning paradigm (Weller et al., 2020), where language models learn from human instructions to perform unseen tasks in zero-shot scenarios, has received increasing attention re- cently. Compared to conventional machine learn- ing paradigms that mainly learn from data exam- ples, instruction learning requires models to com- plete tasks based on the understanding of human- written task descriptions without task-specific data, † Corresponding author. 100 101 102 103 104 Number of Labeled Data in Each Task 44 46 48 50 52 54 56 58 Performance of Instruction Tuning T0 3B (Sanh et al., 2022) UDIT 700M 304 Instr. IT 700M 304 Instr. IT 700M 154 Instr. IT 700M 46 Instr. Figure 1: The performance of IT and UDIT with respect to the instruction numbers and labeled data amounts. We follow Sanh et al. (2022) to fine-tune a 700M PLM and then test its zero-shot generalization ability on un- seen tasks. The x-axis represents the number of labeled samples in each training task, and the y-axis is the av- erage performance on evaluation tasks. We control the instruction numbers by gradually adding training tasks. which is closer to general AI systems. For instance, in summarization tasks, a model is only given an explicit instruction “Summarize the following …

Content truncated to 500 words