Crosslingual Generalization through Multitask Finetuning

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 15991–16111 July 9-14, 2023 ©2023 Association for Computational Linguistics Crosslingual Generalization through Multitask Finetuning Niklas Muennighoff1 Thomas Wang1 Lintang Sutawika2,3 Adam Roberts4 Stella Biderman3,5 Teven Le Scao1 M Saiful Bari6 Sheng Shen7 Zheng-Xin Yong8 Hailey Schoelkopf 3,9 Xiangru Tang 9 Dragomir Radev9 Alham Fikri Aji10 Khalid Almubarak11 Samuel Albanie12 Zaid Alyafeai13 Albert Webson8 Edward Raff5 Colin Raffel1 1Hugging Face 2Datasaur.ai 3EleutherAI 4Google Research, Brain Team 5Booz Allen Hamilton 6NTU 7 UC Berkeley 8 Brown University 9 Yale University 10 MBZUAI 11 PSAU 12 University of Cambridge 13 KFUPM niklas@hf.co Abstract Multitask prompted finetuning (MTF) has been shown to help large language models gener- alize to new tasks in a zero-shot setting, but so far explorations of MTF have focused on English data and models. We apply MTF to the pretrained multilingual BLOOM and mT5 model families to produce finetuned variants called BLOOMZ and mT0. We find finetuning large multilingual language models on English tasks with English prompts allows for task gen- eralization to non-English languages that ap- pear only in the pretraining corpus. Finetun- ing on multilingual tasks with English prompts further improves performance on English and non-English tasks leading to various state-of- the-art zero-shot results. We also investigate finetuning on multilingual tasks with prompts that have been machine-translated from En- glish to match the language of each dataset. We find training on these machine-translated prompts leads to better performance on human- written prompts in the respective languages. Surprisingly, we find models are capable of zero-shot generalization to tasks in languages they have never intentionally seen. We conjec- ture that the models are learning higher-level capabilities that are both task- and language- agnostic. In addition, we introduce xP3, a composite of supervised datasets in 46 lan- guages with English and machine-translated prompts. Our code, datasets and models are freely available at https://github.com/ bigscience-workshop/xmtf. 1 Introduction Large language models pretrained on vast amounts of text show some capability of solving tasks ex- pressed in natural language, even without explicit training on these tasks (Brown et al., 2020). Fine- tuning on groups of language tasks has been shown to significantly boost this zero-shot task general- ization of language models (Wei et al., 2021; Sanh et al., 2022; Min et al., 2021). For example, Sanh et al. (2022) finetune on tasks like summarization and question answering leading to better perfor- mance on unseen tasks like natural language in- ference. Previous work has focused on multitask finetuning in the context of large English language models and tasks. Multilingual large language models show the same zero-shot learning capabilities for both mono- lingual and crosslingual tasks (Goyal et al., 2021a; Lin et al., 2021; Patel et al., 2022; Soltan et al., 2022). However, zero-shot performance tends to be significantly lower than finetuned performance. Thus, task-specific or language-specific transfer learning via finetuning remains the predominant practice (Devlin et al., 2018; Conneau et al., 2019; Aribandi et al., 2021). This is …

Content truncated to 500 words