On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5168 - 5186 July 10-15, 2022 ©2022 Association for Computational Linguistics On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model Seongjin Shin∗,1 Sang-Woo Lee∗,1,2 Hwijeen Ahn1 Sungdong Kim2 HyoungSeok Kim1 Boseop Kim1 Kyunghyun Cho3 Gichang Lee1 Woomyoung Park1 Jung-Woo Ha1,2 Nako Sung1 NAVER CLOVA1 NAVER AI Lab2 NYU3 Abstract Many recent studies on large-scale language models have reported successful in-context zero- and few-shot learning ability. However, the in-depth analysis of when in-context learn- ing occurs is still lacking. For example, it is unknown how in-context learning performance changes as the training corpus varies. Here, we investigate the effects of the source and size of the pretraining corpus on in-context learn- ing in HyperCLOVA, a Korean-centric GPT- 3 model. From our in-depth investigation, we introduce the following observations: (1) in- context learning performance heavily depends on the corpus domain source, and the size of the pretraining corpus does not necessarily de- termine the emergence of in-context learning, (2) in-context learning ability can emerge when a language model is trained on a combination of multiple corpora, even when each corpus does not result in in-context learning on its own, (3) pretraining with a corpus related to a downstream task does not always guarantee the competitive in-context learning performance of the downstream task, especially in the few-shot setting, and (4) the relationship between lan- guage modeling (measured in perplexity) and in-context learning does not always correlate: e.g., low perplexity does not always imply high in-context few-shot learning performance. 1 Introduction NLP community has been surprised by emergence of in-context learning ability of a large-scale lan- guage model (LM) such as GPT-3 (Brown et al., 2020) despite no duplication between downstream task data and the pretraining corpus. Indeed, in- context learning uses a natural language descrip- tion and a few examples to prime a language model. Then the language model can predict the answer of a new example without updating the language model’s parameters. Since the release of GPT-3, ∗Equal contribution. various large-scale in-context language models have been proposed (Black et al., 2021; Kim et al., 2021; Zeng et al., 2021; Rae et al., 2021; Hoffmann et al., 2022; Chowdhery et al., 2022). There still remain many questions on language models’ in-context learning capability despite these successful reports. For example, the relationship between the choice of a pretraining corpus and downstream in-context learning task accuracy is un- known. Previous studies argue pretraining with the corpus similar to the downstream task improves the downstream performance, but these observations are often limited to the case where a pretrained lan- guage model is finetuned for the downstream task (Gururangan et al., 2020; Lee et al., 2020; Micheli et al., 2020). In addition, analysis on the relation between the validation perplexity of a language model and in- context learning performance is still less investi- gated. Previous research on in-context learning im- plicitly assumes …

Content truncated to 500 words