How Do In-Context Examples Affect Compositional Generalization?
38070Authors
nan
Abstract
AbstractCompositional generalization–understanding unseen combinations of seen primitives–is an essential reasoning capability in human intelligence. The AI community mainly studies this capability by fine-tuning neural networks on lots of training samples, while it is still unclear whether and how in-context learning–the prevailing few-shot paradigm based on large language models–exhibits compositional generalization. In this paper, we present CoFe, a test suite to investigate in-context compositional generalization. We find that the compositional generalization performance can be easily affected by the selection of in-context examples, thus raising the research question what the key factors are to make good in-context examples for compositional generalization. We study three potential factors: similarity, diversity and complexity. Our systematic experiments indicate that in-context examples should be structurally similar to the test case, diverse from each other, and individually simple. Furthermore, two strong limitations are observed: in-context compositional generalization on fictional words is much weaker than that on commonly used ones; it is still critical that the in-context examples should cover required linguistic structures, even though the backbone model has been pre-trained on large corpus. We hope our analysis would facilitate the understanding and utilization of in-context learning paradigm.
Keywords
Full Content
Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 1082–1117 July 9-14, 2023 ©2023 Association for Computational Linguistics Glot500: Scaling Multilingual Corpora and Language Models to 500 Languages Ayyoob Imani∗1,2, Peiqin Lin∗1,2, Amir Hossein Kargaran1,2, Silvia Severini1, Masoud Jalili Sabet1, Nora Kassner1,2, Chunlan Ma1,2, Helmut Schmid1, André F. T. Martins3,4,5, François Yvon6 and Hinrich Schütze1,2 1CIS, LMU Munich, Germany 2Munich Center for Machine Learning (MCML), Germany 3Instituto Superior Técnico (Lisbon ELLIS Unit) 4Instituto de Telecomunicações 5Unbabel 6Sorbonne Université, CNRS, ISIR, France {ayyoob, linpq, amir, silvia}@cis.lmu.de Abstract The NLP community has mainly focused on scaling Large Language Models (LLMs) ver- tically, i.e., making them better for about 100 languages. We instead scale LLMs horizon- tally: we create, through continued pretraining, Glot500-m, an LLM that covers 511 predom- inantly low-resource languages. An impor- tant part of this effort is to collect and clean Glot500-c, a corpus that covers these 511 lan- guages and allows us to train Glot500-m. We evaluate Glot500-m on five diverse tasks across these languages. We observe large improve- ments for both high-resource and low-resource languages compared to an XLM-R baseline. Our analysis shows that no single factor ex- plains the quality of multilingual LLM rep- resentations. Rather, a combination of fac- tors determines quality including corpus size, script, “help” from related languages and the total capacity of the model. Our work ad- dresses an important goal of NLP research: we should not limit NLP to a small fraction of the world’s languages and instead strive to support as many languages as possible to bring the ben- efits of NLP technology to all languages and cultures. Code, data and models are available at https://github.com/cisnlp/Glot500. 1 Introduction The NLP community has mainly focused on scaling Large Language Models (LLMs) vertically, i.e., deepening their understanding of high-resource lan- guages by scaling up parameters and training data. While this approach has revolutionized NLP, the achievements are largely limited to high-resource languages. Examples of “vertical” LLMs are GPT3 (Brown et al., 2020), PaLM (Chowdhery et al., 2022) and Bloom (BigScience et al., 2022). In this paper, we create Glot500-m, a model that instead focuses on scaling multilingual LLMs horizontally, i.e., scaling to a large number of languages the great *Equal contribution. majority of which is low-resource. As LLMs are essential for progress in NLP, lack of LLMs support- ing low-resource languages is a serious impediment to bringing NLP to all of the world’s languages and cultures. Our goal is to address this need with the creation of Glot500-m.1 Existing multilingual LLMs support only about 100 (Conneau et al., 2020) out of the 7000 languages of the world. These supported languages are the ones for which large amounts of training data are available through projects such as Oscar (Suárez et al., 2019) and the Wikipedia dumps.2 Following Siddhant et al. (2022), we refer to the 100 languages covered by XLM-R (Conneau et al., 2020) as head languages and to the remaining languages as tail languages. This …
Content truncated to 500 words
Labeled Text
AbstractCompositional generalization–understanding unseen combinations of seen primitives–is an essential reasoning capability in human intelligence. The AI community mainly studies this capability by fine-tuning neural networks on lots of training samples, while it is still unclear whether and how in-context learning–the prevailing few-shot paradigm based on large language models–exhibits compositional generalization. In this paper, we present CoFe, a test suite to investigate in-context compositional generalization. We find that the compositional generalization performance can be easily affected by the selection of in-context examples, thus raising the research question what the key factors are to make good in-context examples for compositional generalization. We study three potential factors: similarity, diversity and complexity. Our systematic experiments indicate that in-context examples should be structurally similar to the test case, diverse from each other, and individually simple. Furthermore, two strong limitations are observed: in-context compositional generalization on fictional words is much weaker than that on commonly used ones; it is still critical that the in-context examples should cover required linguistic structures, even though the backbone model has been pre-trained on large corpus. We hope our analysis would facilitate the understanding and utilization of in-context learning paradigm.
Publication Details
- Paper ID:
- 38070
- Year:
- 2023
- Publication:
- TIPSTER
- Publication Type:
- conf
- DOI:
- nan