Evolving Large Text Corpora: Four Versions of the Icelandic Gigaword Corpus

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 2371–2381 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 2371 Evolving Large Text Corpora: Four Versions of the Icelandic Gigaword Corpus Starkaður Barkarson, Steinþór Steingrímsson, Hildur Hafsteinsdóttir The Árni Magnússon Institute for Icelandic Studies {starkadur.barkarson, steinthor.steingrimsson, hildur.hafsteinsdottir}@arnastofnun.is Abstract The Icelandic Gigaword Corpus was first published in 2018. Since then new versions have been published annually, containing new texts from additional sources as well as from previous sources. This paper describes the evolution of the corpus in its first four years. All versions are made available under permissive licenses and with each new version the texts are annotated with the latest and most accurate tools. We show how the corpus has grown almost 50% in size from the first version to the fourth and how it was restructured in order to better accommodate different meta-data for different subcorpora. Furthermore, other services have been set up to facilitate usage of the corpus for different use cases. These include a keyword-in-context concordance tool, an n-gram viewer, a word frequency database and pre-trained word embeddings. Keywords: text corpora, Icelandic, gigaword 1. Introduction Large text corpora are useful in various fields of research. They can be used to study language variation (Iomdin et al., 2013), for compiling dictionaries, see e.g. Sinclair (1987), Gizatova (2016), Jónsdóttir and Úlfarsdóttir (2020), and for developing NLP applications. Since the advent of Word2Vec (Mikolov et al., 2013), static word embeddings trained using neural networks, various neural network language models have been employed for most if not all common NLP tasks. These language models require large corpora of target language texts for training. Although the first word embedding models could produce meaningful results with only a few million tokens in the training data (Jungmaier et al., 2020), for best results they are trained on corpora containing hundreds of millions or even billions of words (Grave et al., 2018). While ever larger language models are being developed, with GPT-3 (Brown et al., 2020) trained on approximately 500 billion tokens and Gopher (Rae et al., 2021) using two trillion words for training, the more common BERT-like models (Devlin et al., 2019) of contextualized embeddings, traditionally trained on multibillion token corpora, have been shown to give very competitive results when trained on only approximately one billion tokens, see e.g. Martin et al. (2020). The need for large and updated text corpora is thus evident if a language is to be a part of the development of language technology (LT). In 2019, a national language technology programme for Icelandic (Nikulásdóttir et al., 2020) was started. As a part of that programme, the Icelandic Giga- word Corpus (IGC) (Steingrímsson et al., 2018) was to be enlarged and evolved, with new versions published every year. This paper describes the evolution of the corpus from its initial version, published in 2018, to the fourth version, published in 2021. We describe the content, annotation pro- cess and licensing, the division of …

Content truncated to 500 words