Local Languages, Third Spaces, and other High-Resource Scenarios

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 719 - 730 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Meta-learning via Language Model In-context Tuning Yanda Chen1∗ Ruiqi Zhong2 Sheng Zha3 George Karypis3 He He34 1Columbia University, 2University of California, Berkeley, 3AWS AI 4New York University yc3384@columbia.edu, ruiqi-zhong@berkeley.edu, {zhasheng, gkarypis, hehea}@amazon.com Abstract The goal of meta-learning is to learn to adapt to a new task with only a few labeled examples. Inspired by the recent progress in large lan- guage models, we propose in-context tuning (ICT), which recasts task adaptation and pre- diction as a simple sequence prediction prob- lem: to form the input sequence, we concate- nate the task instruction, labeled in-context ex- amples, and the target input to predict; to meta- train the model to learn from in-context ex- amples, we fine-tune a pre-trained language model (LM) to predict the target label given the input sequence on a collection of tasks. We benchmark our method on two collections of text classification tasks: LAMA and Bina- ryClfs. Compared to MAML which adapts the model through gradient descent, our method leverages the inductive bias of pre-trained LMs to perform pattern matching, and out- performs MAML by an absolute 6% average AUC-ROC score on BinaryClfs, gaining more advantage with increasing model size. Com- pared to non-fine-tuned in-context learning (i.e. prompting a raw LM), in-context tuning meta-trains the model to learn from in-context examples. On BinaryClfs, ICT improves the average AUC-ROC score by an absolute 10%, and reduces the variance due to example order- ing by 6x and example choices by 2x. 1 Introduction Few-shot learning (FSL) refers to a system’s ability to quickly adapt to new tasks when very few labeled examples are available for training. FSL is a key feature of human learning (Lake et al., 2016), but current machine learning systems often rely on large amounts of labeled training data (Silver et al., 2016; He et al., 2016; Adiwardana et al., 2020). Recently, prompting large pre-trained language models (LMs) for FSL has achieved remarkable progress (Brown et al., 2020; Schick and Schütze, ∗ Work done during summer internship at AWS AI. 2021a). LM prompting with in-context learning reduces the “task learning and predict” process to a simple sequence prediction problem. To perform a new task, Brown et al. (2020) prompt a raw LM (i.e., a pre-trained LM not fine-tuned on any labeled data) with the concatenation of the task instruction, some input-output examples, and the target input to be predicted on; then they extract the answer from the LM’s continuation of the concatenated sequence (Figure 1 left). For example, to coax the model into performing sentiment classification on the target input “This movie is a waste of time”, we prompt the LM with the sequence “I like the movie! Positive review? Yes. Horrible Movie! Positive review? No. This movie is a waste of time. Positive review? ___”, and predict “positive” if the next word is more likely to be “Yes” …

Content truncated to 500 words