Large-scale Lifelong Learning of In-context Instructions and How to Tackle It

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 12573–12589 July 9-14, 2023 ©2023 Association for Computational Linguistics Large-scale Lifelong Learning of In-context Instructions and How to Tackle It Jisoo Mok1∗ Jaeyoung Do2† Sungjin Lee2 Tara Taghavi2 Seunghak Yu3 Sungroh Yoon1,4† 1 Department of ECE, Seoul National University 2 Amazon Alexa AI 3 NAVER Search US 4 Interdisciplinary Program in AI, Seoul National University Abstract Jointly fine-tuning a Pre-trained Language Model (PLM) on a pre-defined set of tasks with in-context instructions has been proven to improve its generalization performance, al- lowing us to build a universal language model that can be deployed across task boundaries. In this work, we explore for the first time whether this attractive property of in-context instruction learning can be extended to a scenario in which tasks are fed to the target PLM in a sequen- tial manner. The primary objective of so-called lifelong in-context instruction learning is to im- prove the target PLM’s instance- and task-level generalization performance as it observes more tasks. DYNAINST, the proposed method to life- long in-context instruction learning, achieves noticeable improvements in both types of gen- eralization, nearly reaching the upper bound performance obtained through joint training. 1 Introduction A number of recent studies have shown that Pre- trained Language Models (PLMs) only need to undergo a brief fine-tuning process to achieve remarkable instance-level generalization perfor- mance within the observed task (Brown et al., 2020). Compared to instance-level generalization within seen tasks, however, the zero-shot cross-task generalization capability of PLMs to unseen tasks is just starting to be explored (Sanh et al., 2021; Wei et al., 2021). In order to build a language model that can generalize across task boundaries and thus be deployed to various task scenarios, in-context instruction learning draws inspiration from how hu- mans can familiarize themselves with a variety of language-related tasks only by following instruc- tions (Mishra et al., 2022). To emulate the human learning process, in-context instruction learning trains the target PLM with both input-output pairs ∗Work done while interning at Amazon Alexa AI (magic- shop1118@snu.ac.kr) †Corresponding Authors and a set of in-context instructions that contain additional task-specific information. The introduction of in-context instructions is spearheading a noticeable progress in cross-task generalization within Natural Language Processing (NLP) research. The recent advances in-context instruciton learning (Ouyang et al., 2022; Mishra et al., 2022; Wang et al., 2022) indicate that fine- tuning a PLM with task-specific instructions has a positive impact on its cross-task generalization ca- pability. One common limitation shared by the ex- isting works is that they require a static, pre-defined set of tasks to jointly train the target PLM. Any learning paradigm bound by the assumption that all of the tasks are pre-defined and non-changing not only incurs a huge memory and computa- tional cost but also raises serious data privacy con- cerns (De Lange et al., 2021). In this paper, we aim to address such concerns by studying whether it is possible to sequentially …

Content truncated to 500 words