The Norwegian Colossal Corpus: A Text Corpus for Training Large Norwegian Language Models

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 3852–3860 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 3852 The Norwegian Colossal Corpus: A Text Corpus for Training Large Norwegian Language Models Per E Kummervold, Freddy Wetjen, Javier de la Rosa National Library of Norway (NLN), Norway per@capia.no, freddy.wetjen@nb.no, versae@nb.no Abstract Norwegian has been one of many languages lacking sufficient available text to train quality language models. In an attempt to bridge this gap, we introduce the Norwegian Colossal Corpus (NCC), which comprises 49GB of clean Norwegian textual data containing over 7B words. The NCC is composed of different and varied sources, ranging from books and newspapers to government documents and public reports, showcasing the various uses of the Norwegian language in society. The corpus contains mainly Norwegian Bokmål and Norwegian Nynorsk. Each document in the corpus is tagged with metadata that enables the creation of sub-corpora for specific needs. Its structure makes it easy to combine with large web archives that for licensing reasons could not be distributed together with the NCC. By releasing this corpus openly to the public, we hope to foster the creation of both better Norwegian language models and multilingual language models with support for Norwegian. Keywords: natural language, text corpus, Norwegian 1. Introduction Most modern approaches to natural language processing (NLP) require the training of language models on vast amounts of text. Ideally, these textual sources represent the language as it is used in a variety of settings. The power of unsupervised training on extremely large corpora was first demonstrated by Devlin et al. (2019) in their seminal work Bidirectional Encoder Representation from Transformers (BERT). Funded by Google, their model was trained on 16GB of uncompressed English text and comprised approximately 3.3B words. As the most popular language online, the size of the available corpora for English has steadily increased to over 800GB of text (Gao et al., 2021), a figure that is difficult to reach for lower-resource languages . 1 With only about 5 million native speakers, Norwegian is considered one of the lower-resource languages. The Norwegian language is a single spoken language with two official written languages: Bokmål and Nynorsk. Bokmål is the predominant written form and is the most widely used in everyday life in Norway. However, public entities are required by the Norwegian Language Act [Språklova] to produce at least a quarter of their publicly available documents in Nynorsk (Lovdata, 2022). 1.1 Norwegian Transformer Models Until recently, the only way to utilize language models with languages such as Norwegian was via multilingual models. The multilingual version of BERT (mBERT) (Devlin et al., 2019) was trained solely on Wikipedia dumps constituting 104 different languages. Two of those languages were Norwegian Bokmål and Norwegian Nynorsk. While they did not specify the exact size of the Norwegian text in the corpus, our estimate suggests it was between 0.5GB and 1.0GB of text (70M and 140M words, respectively), where roughly 80% was Norwegian Bokmål. A …

Content truncated to 500 words