Lessons Learned from GPT-SW3: Building the First Large-Scale Generative Language Model for Swedish

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 3509–3518 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 3509 Lessons Learned from GPT-SW3: Building the First Large-Scale Generative Language Model for Swedish Ariel Ekgren⋆, Amaru Cuba Gyllensten◦, Evangelia Gogoulou◦, Alice Heiman⋆, Severine Verlinden⋆, Joey ¨Ohman⋆, Fredrik Carlsson◦, Magnus Sahlgren⋆ ⋆AI Sweden, ◦RISE {ariel.ekgren, alice.heiman, severine.verlinden, joey.ohman, magnus.sahlgren}@ai.se {amaru.cuba.gyllensten, evangelia.gogoulou, fredrik.carlsson}@ri.se Abstract We present GPT-SW3, a 3.5 billion parameter autoregressive language model, trained on a newly created 100 GB Swedish corpus. This paper provides insights with regard to data collection and training process, and discusses the challenges of proper evaluation. The results of quantitive evaluation using perplexity indicate that GPT-SW3 is a competent model in comparison with existing autoregressive models of similar size. Additionally, we perform an extensive prompting study which reveals the good text generation capabilities of GPT-SW3. Keywords: Language models, Evaluation, Prompting 1. Introduction Large-scale generative language models such as the GPT series (Radford and Narasimhan, 2018; Radford et al., 2019; Brown et al., 2020) have enjoyed consider- able attention in recent years. This has been partly due to their unprecedented ability to generate coherent text, but also for their capacity for zero-shot performance - without any training examples, on a wide range of different tasks. A prerequisite for building such mod- els is access to both large amounts of high-quality text data and powerful computational resources. This has proven to be a limiting factor for the development of large-scale models for languages other than English. With the goal of promoting the development of large- scale generative models for other languages, we here present our work on developing and evaluating GPT- SW3, a 3.5 billion parameter autoregressive language model, trained on a newly collected 100 GB Swedish corpus. To the best of our knowledge, this is the largest generative model for Swedish to date, and probably one of the bigger non-English models at the moment. In this paper, we collect the lessons learned by developing and evaluating this model, including challenges with data collection, training procedures, and validation ac- tivities. 2. Related Work The development of large-scale (i.e. billions of pa- rameters) generative language models has largely been dominated by the GPT series of models, starting with GPT-2 with 1.5 billion parameters (Radford et al., 2019), over Megatron-LM with 8.3 billion parameters (Shoeybi et al., 2019a) and Turing-NLG with 17.2 bil- lion parameters,1 to GPT-3 with 175 billion parameters (Brown et al., 2020). More recently, Nvidia and Mi- crosoft announced a generative model called Megatron- 1https://bit.ly/31425Vc Turing NLG with a whopping 530 billion parameters.2 None of these truly large-scale models (i.e. models with a parameter count over 10 billion parameters) are pub- licly available at this time. One initiative that aims to provide open-source large- scale generative language models is Eleuther AI3, which has so far provided GPT-Neo with 2.7 billion parameters (Black et al., 2021), GPT-J with 6 bil- lion parameters (Wang and Komatsuzaki, 2021) and GPT-NeoX with 20 …

Content truncated to 500 words