A Warm Start and a Clean Crawled Corpus - A Recipe for Good Language Models

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 4356–4366 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 4356 A Warm Start and a Clean Crawled Corpus - A Recipe for Good Language Models Vésteinn Snæbjarnarson1,2,*, Haukur Barri Símonarson1,2,*, Pétur Orri Ragnarsson1, Svanhvít Lilja Ingólfsdóttir1, Haukur Páll Jónsson1, Vilhjálmur Þorsteinsson1, Hafsteinn Einarsson2 1 Miðeind ehf., 2 University of Iceland {vesteinn, haukur, petur, svanhvit, haukurpj, vt}@mideind.is, hafsteinne@hi.is Abstract We train several language models for Icelandic, including IceBERT, that achieve state-of-the-art performance in a variety of downstream tasks, including part-of-speech tagging, named entity recognition, grammatical error detection and constituency parsing. To train the models we introduce a new corpus of Icelandic text, the Icelandic Common Crawl Corpus (IC3), a collection of high quality texts found online by targeting the Icelandic top-level-domain .is. Several other public data sources are also collected for a total of 16GB of Icelandic text. To enhance the evaluation of model performance and to raise the bar in baselines for Icelandic, we manually translate and adapt the WinoGrande commonsense reasoning dataset. Through these efforts we demonstrate that a properly cleaned crawled corpus is sufficient to achieve state-of-the-art results in NLP applications for low to medium resource languages, by comparison with models trained on a curated corpus. We further show that initializing models using existing multilingual models can lead to state-of-the-art results for some downstream tasks. Keywords: language model, Icelandic, IceBERT, corpus, part of speech, named entity recognition, parsing, co-reference resolution, natural language understanding 1. Introduction The Government of Iceland recently launched an ini- tiative to improve the state of Icelandic language re- sources and language technology (Nikulásdóttir et al., 2020). This comprehensive program has its roots in the historical focus on protecting the Icelandic lan- guage (Kristinsson, 2018), and, as a result, work has been ongoing to build and enhance said resources. This effort is gradually pushing Icelandic from being a low- resource language to a medium-resource one1. Still, and apart from large monolingual corpora (Steingríms- son et al., 2018), many important types of resources are lacking in comparison with major languages such as English. Parallel to the development of the Icelandic language technology program, language technology world-wide has been progressing at a fast and accelerating pace (Bommasani et al., 2021). Pre-trained neural language models based on Transformers (Vaswani et al., 2017) have shown impressive results when adapted for a va- riety of classification and text generation tasks. Such models are now applied widely across industries and modalities. Large monolingual language models such as BERT (Devlin et al., 2019), BART (Lewis et al., 2020) and GPT-2 (Radford et al., 2019) have been developed for English. For many smaller languages the only available options are multilingual models, which * Equal contribution. 1Low-resource is not a precise term, but a language can be considered to be low-resource if few online resources exist for it (Cieri et al., 2016). can reach impressive performance on downstream tasks (Conneau et al., 2020). These are not …

Content truncated to 500 words