Risamálheild: A Very Large Icelandic Text Corpus

Risamálheild: A Very Large Icelandic Text Corpus Steinþór Steingrímsson1, Sigrún Helgadóttir1, Eiríkur Rögnvaldsson2, Starkaður Barkarson1, Jón Guðnason3 1The Árni Magnússon Institute for Icelandic Studies, Reykjavík, Iceland 2The University of Iceland, Reykjavík, Iceland 3Reykjavik University, Reykjavík, Iceland steinst@hi.is, sigruhel@hi.is, eirikur@hi.is, stb36@hi.is, jg@ru.is Abstract We present Risamálheild, the Icelandic Gigaword Corpus (IGC), a corpus containing more than one billion running words from mostly contemporary texts. The work was carried out with minimal amount of work and resources, focusing on material that is not protected by copyright and sources which could provide us with large chunks of text for each cleared permission. The two main sources considered were therefore official texts and texts from news media. Only digitally available texts are included in the corpus and formats that can be problematic are not processed. The corpus texts are morphosyntactically tagged and provided with metadata. Processes have been set up for continuous text collection, cleaning and annotation. The corpus is available for search and download with permissive licenses. The dataset is intended to be clearly versioned with the first version released in early 2018. Texts will be collected continually and a new version published every year. Keywords: text corpora, Icelandic 1. Introduction The lack of a very large Icelandic text corpus has been evident for some time. Data oriented methods have increasingly come to dominate the field of NLP and this has led to the need for more data and bigger datasets in order to achieve better performance. In the last few years, with machine learning methods such as neural networks reaching preeminence in various areas of Language Technology (LT) the importance of large text corpora and other textual resources has increased considerably. The compilation of a corpus such as the one described here has therefore been considered a top priority in order to further LT in Iceland (Nikulásdóttir et al. 2017). Large text corpora are e.g. necessary for the design of language models that are used in building a variety of LT tools such as speech recognizers, spell and grammar checkers and automatic machine translation systems. The aim of the IGC project was to compile as large a corpus as possible with the minimum amount of work and resources. The corpus should be clearly versioned in order to facilitate reproducible experiments. The design should make it easy to compare NLP algorithms on contemporary Icelandic and serve as a resource for a wide range of linguistic research, research in the field of culturomics (Michel et al. 2011) and for the interested public. The corpus should be attractive for use in LT projects as well as for other research and study. Therefore we aimed for the following goals: • The IGC will contain more than a billion running words, morphosyntactically tagged and lemmatized and provided with metadata. • Only digitally available texts will be included in the IGC. Formats that may pose a difficulty will not be processed. • The IGC will be open and constantly expanding. • A closed version will be published every …

Content truncated to 500 words