Natural Questions in Icelandic

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 4488–4496 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 4488 Natural Questions in Icelandic Vésteinn Snæbjarnarson1,2, Hafsteinn Einarsson2 1Miðeind ehf, 2University of Iceland vesteinn@mideind.is, hafsteinne@hi.is Abstract We present the first extractive question answering (QA) dataset for Icelandic, Natural Questions in Icelandic (NQiI). Developing such datasets is important for the development and evaluation of Icelandic QA systems. It also aids in the development of QA methods that need to work for a wide range of morphologically and grammatically different languages in a multilingual setting. The dataset was created by asking contributors to come up with questions they would like to know the answer to. Later, they were tasked with finding answers to each others questions following a previously published methodology. The questions are Natural in the sense that they are real questions posed out of interest in knowing the answer. The complete dataset contains 18 thousand labeled entries of which 5,568 are directly suitable for training an extractive QA system for Icelandic. The dataset is a valuable resource for Icelandic which we demonstrate by creating and evaluating a system capable of extractive QA in Icelandic. Keywords: QA, question answering, Icelandic 1. Introduction Most of us use QA systems daily through the use of search engines. Generally, the user asking a question wants to know, in their formulation, the answer to the question and its location within a source document. To build, or at the very least, validate such systems it is currently necessary to have a properly labeled dataset in the underlying language. Such datasets contain ques- tions and underlying context, along with labelled an- swer spans within the context. Several datasets exist for this task in English, such as WikiQA (Yang et al., 2015), and the Natural Questions (NQ) dataset (Kwiatkowski et al., 2019). The need to create similar datasets in other languages was recog- nized recently (Clark et al., 2020). While English is spoken by many, there exist thousands of languages and most of the world’s population does not speak En- glish. Furthermore, languages differ widely in typol- ogy such as grammar and morphology. All these dif- ferences might affect a QA system’s performance. To study the effects of language differences on QA sys- tems, such a QA dataset on 11 typologically diverse languages was released by Clark et al. (2020). Other such datasets have also been created for a variety of languages (Rogers et al., 2021) as they are a prerequi- site for accurate evaluation of QA systems for a specific language. In this paper, we describe and release Natural Ques- tions in Icelandic (NQiI), the first dataset for boolean (questions with yes-no answers) and extractive QA in Icelandic that has not been scraped from the web. The dataset contains around 13.7k questions and 18k question-passage pairs with roughly 5k containing an- swers. We used the methodology presented by Clark et al. (2020) to create questions that reflect information- seeking …

Content truncated to 500 words