Evaluation Set for Slovak News Information Retrieval

Evaluation Set for Slovak News Information Retrieval Daniel Hl´adek, Jan Staˇs, Jozef Juh´ar Department of Electronics and Multimedia Communications Technical University of Koˇsice, Slovak Republic daniel.hladek@tuke.sk, jan.stas@tuke.sk, jozef.juhar@tuke.s Abstract This work proposes an information retrieval evaluation set for the Slovak language. A set of 80 queries written in the natural language is given together with the set of relevant documents. The document set contains 3980 newspaper articles sorted into 6 categories. Each doc- ument in the result set is manually annotated for relevancy with its corresponding query. The evaluation set is mostly compatible with the Cranfield test collection using the same methodology for queries and annotation of relevancy. In addition to that it provides annotation for document title, author, publication date and category that can be used for evaluation of automatic document clustering and categorization. Keywords: information retrieval evaluation, Cranfield test collection, Slovak language 1. Introduction In order to accelerate research of information retrieval (IR) techniques for less resourced languages we present an eval- uation benchmark consisting of a document set, queries and relevant documents. This paper proposes a method to eval- uate information retrieval system for a morphologically rich language - Slovak. Usually it is hard to evaluate a design of an IR system for a language different from English because of the lack of properly annotated databases, especially if number of speakers of the language is lower. Importance of IR for non-English languages rise together with number and size of national Internets and amount of human-entered textual data in business and government databases (Korra et al., 2011; Lazarinis et al., 2009). Most problems in IR are language independent, such as document representations, clustering or classification. On the other hand, there are several issues that are bound to a specific language and have to be solved in order to build a successful IR system. This contribution aims to fill this gap and helps to evaluate techniques that are adapted to a non-English language. Slovak language has lesser num- ber of language-specific resources available that are neces- sary to build an IR system, such as WordNet, morpholog- ical analysis tools or vocabularies. Languages with a rich morphology and free word order usually require specialized methods for stem identification, word and sentence bound- ary detection or different features for chunking or named entity recognition. 2. Previous Works Most of the research in the field of information retrieval is focused on the English language. There is a large amount of evaluation benchmarks for IR in English. The most basic is the Cranfield collection based on work (Cleverdon, 1967). It contains a set of information needs from a database of ab- stracts. TREC (Simpson et al., 2014) and CLEF (Suominen et al., 2014) are the biggest series of evaluation campaigns focused on various tasks of IR. Multi-lingual (Peters et al., 2012) and cross-lingual IR are gaining a lot of attention, but most of the current evaluation databases contain just couple of the most commonly used languages such as Chi- nese or French. There …

Content truncated to 500 words