Efficient Large Scale Language Modeling with Mixtures of Experts

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11699 - 11732 December 7-11, 2022 ©2022 Association for Computational Linguistics Few-shot Learning with Multilingual Generative Language Models Xi Victoria Lin*, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyanov, Xian Li* Meta AI Abstract Large-scale generative language models such as GPT-3 are competitive few-shot learners. While these models are known to be able to jointly represent multiple languages, their train- ing data is dominated by English, potentially limiting their cross-lingual generalization. In this work, we train multilingual generative lan- guage models on a corpus covering a diverse set of languages, and study their few- and zero- shot learning capabilities in a wide range of tasks. Our largest model with 7.5 billion pa- rameters sets new state of the art in few-shot learning in more than 20 representative lan- guages, outperforming GPT-3 of comparable size in multilingual commonsense reasoning (with +7.4% absolute accuracy improvement in 0-shot settings and +9.4% in 4-shot set- tings) and natural language inference (+5.4% in each of 0-shot and 4-shot settings). On the FLORES-101 machine translation benchmark, our model outperforms GPT-3 counterparts on 171 out of 182 directions with 32 training ex- amples, while surpassing the official super- vised baseline in 45 directions. We conduct an in-depth analysis of different multilingual prompting approaches, showing in particular that strong in-context few-shot learning perfor- mance across languages can be achieved via cross-lingual transfer through both templates and demonstration examples.1 1 Introduction Large autoregressive language models such as GPT-3 can be adapted, via few- and zero-shot learning, to a wide range of tasks with signifi- cantly less cost than full fine-tuning (Brown et al., 2020; Bommasani et al., 2021). These models have been primarily developed for English. Although * Equal contribution. Correspondence to: ⟨victorialin@meta.com, xianl@meta.com⟩. 1Our checkpoints, code and new dataset (XStoryCloze): https://github.com/facebookresearch/ fairseq/tree/main/examples/xglm. the training data of GPT-3 contains a small per- centage of non-English text (7%) allowing it to achieve some promising cross-lingual generaliza- tion, the model is almost exclusively deployed for use cases in English. Multilingual masked and sequence-to-sequence language models have been studied, including mBERT, XLM-R, mT5, and mBART (Devlin et al., 2019; Conneau et al., 2020; Xue et al., 2020; Fedus et al., 2021; Goyal et al., 2021a; Liu et al., 2020). These models are typi- cally fine-tuned on large amount of labeled data in downstream tasks. Despite notable recent work at smaller scales (Zhao and Sch¨utze, 2021) and for domain-specific tasks (Winata et al., 2021), the multilingual few-shot learning capabilities of lan- guage models are less well understood. In this paper, we train four multilingual genera- tive language models (up to 7.5 billion parameters), XGLM’s, and present a comprehensive study of multilingual zero- and in-context few-shot learning. We train the models using a large-scale corpus of 500B tokens that comprises …

Content truncated to 500 words