BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Prompting

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 11682–11703 July 9-14, 2023 ©2023 Association for Computational Linguistics BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Prompting Zheng-Xin Yong1∗, Hailey Schoelkopf2,3, Niklas Muennighoff4, Alham Fikri Aji5, David Ifeoluwa Adelani6, Khalid Almubarak7, M Saiful Bari8, Lintang Sutawika2,9, Jungo Kasai10, Ahmed Baruwa11, Genta Indra Winata12, Stella Biderman2,13, Edward Raff13, Dragomir Radev3, Vassilina Nikoulina14 1Brown University 2EleutherAI 3Yale University 4Hugging Face 5MBZUAI 6University College London 7PSAU 8Nanyang Technological University 9Datasaur.ai 10Paul G. Allen School of CSE, University of Washington 11University of Oregon 12The Hong Kong University of Science and Technology 13Booz Allen Hamilton 14NAVER LABS Europe Abstract The BLOOM model is a large publicly available multilingual language model, but its pretraining was limited to 46 languages. To extend the benefits of BLOOM to other languages without incurring prohibitively large costs, it is desirable to adapt BLOOM to new languages not seen during pretraining. In this work, we apply existing language adaptation strategies to BLOOM and benchmark its zero-shot prompting performance on eight new languages in a resource-constrained setting. We find language adaptation to be effective at improving zero-shot performance in new languages. Surprisingly, we find that adapter-based finetuning is more effective than continued pretraining for large models. In addition, we discover that prompting performance is not significantly affected by language specifics, such as the writing system. It is primarily determined by the size of the language adaptation data. We also add new languages to BLOOMZ, which is a multitask finetuned version of BLOOM capable of following task instructions zero-shot. We find including a new language in the multitask fine-tuning mixture to be the most effective method to teach BLOOMZ a new language. We conclude that with sufficient training data lan- guage adaptation can generalize well to diverse languages. Our code is available at https: //github.com/bigscience-workshop/ multilingual-modeling. 1 Introduction Although access to transformer-based language models has expanded greatly over the past several years (Black et al., 2021; Wang and Komatsuzaki, 2021; Artetxe et al., 2021; Black et al., 2022; Zhang et al., 2022), these technologies are over- whelmingly concentrated in a few high resource ∗Corresponding author: contact.yong@brown.edu languages (Talat et al., 2022). BLOOM (Scao et al., 2022), the largest publicly available multilingual language model to date with 176B parameters, cov- ers only 46 natural languages and even excludes high-resource languages such as Korean and Rus- sian which has tens of millions of speakers. This limitation was driven by a number of factors, most notably only considering languages for which the community had enough expertise to manually vali- date the data quality (Kreutzer et al., 2022), dedu- plicate and remove personally identifiable infor- mation (Laurençon et al., 2022) and had sufficient access to licensed unlabeled text (Joshi et al., 2020). All of these factors are contingent facts about the group that trained the model, and leave open the idea that other researchers could contribute more languages. As regularly retraining such a model is prohibitively expensive, the question …

Content truncated to 500 words