PAGnol: An Extra-Large French Generative Model

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 4275–4284 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 4275 PAGnol: An Extra-Large French Generative Model Julien Launay∗1,2 E.L. Tommasone∗1 Baptiste Pannier1 Franc¸ois Boniface† Am´elie Chatelain1 Alessandro Cappelli1 Iacopo Poli1 Djam´e Seddah3 1 LightOn 2 LPENS, ´Ecole Normale Sup´erieure 3 Inria, Paris {julien,luca,baptiste,amelie,alessandro,iacopo}@lighton.ai djame.seddah@inria.fr Abstract Access to large pre-trained models of varied architectures, in many different languages, is central to the democratization of NLP. We introduce PAGnol, a collection of French GPT models. Using scaling laws, we efficiently train PAGnol-XL (1.5B parameters) with the same computational budget as CamemBERT, a model 13 times smaller. PAGnol-XL is the largest model trained from scratch for the French language. We plan to train increasingly large and performing versions of PAGnol, exploring the capabilities of French extreme-scale models. For this first release, we focus on the pre-training and scaling calculations underlining PAGnol. We fit a scaling law for compute for the French language, and compare it with its English counterpart. We find the pre-training dataset significantly conditions the quality of the outputs, with common datasets such as OSCAR leading to low-quality and offensive text. We evaluate our models on discriminative and generative tasks in French, comparing to other state-of-the-art French and multilingual models, and reaching the state of the art in the abstract summarization task. Our research was conducted on the public GENCI Jean Zay supercomputer, and our models up to the Large are publicly available. Keywords: generative pretrained transformer, scaling laws, French language 1. Introduction Large pre-trained language models are the workhorses of modern Natural Language Processing (NLP). The use of scalable and efficient attention-based Transform- ers (Vaswani et al., 2017), rather than recurrent neural networks, has enabled increasingly large and capable models. Through self-supervised learning, these mod- els learn contextual word embeddings, building a gen- eral representation of language. After this pre-training they can be fine-tuned to target specific tasks (e.g. clas- sification, parsing, summarization). Three approaches dominate the field: (1) causal au- toregressive decoder-only models, such as GPT (Rad- ford et al., 2018), learning from a general language modelling tasks; (2) bidirectional encoder-only mod- els, such as BERT (Devlin et al., 2018), learning from masked language modelling; (3) sequence-to-sequence models, such as BART (Lewis et al., 2019) or T5 (Raf- fel et al., 2020), combining both a bidirectional en- coder and an autoregressive decoder, learning from a language denoising task. Encoder-only and sequence- to-sequence models excel in language understanding tasks, and have shadowed autoregressive models as a lesser option. Autoregressive models have been shown to predictably benefit from increased size (Kaplan et al., 2020; Henighan et al., 2020). Scaling laws establish a di- rect relationship between model size and end-task per- formance, justifying the training of increasingly large models (Brown et al., 2020; Zeng et al., 2021; Kim et al., 2021; Wei et al., 2021). These laws can also in- form design decisions, helping practitioners use their available compute budget optimally. Larger models …

Content truncated to 500 words