New Inflectional Lexicons and Training Corpora for Improved Morphosyntactic Annotation of Croatian and Serbian

New Inflectional Lexicons and Training Corpora for Improved Morphosyntactic Annotation of Croatian and Serbian Nikola Ljubeˇsi´c∗ Filip Klubiˇcka∗ ˇZeljko Agi´c† Ivo-Pavao Jazbec‡ * Dept. of Information and Communication Sciences, University of Zagreb Ivana Luˇci´ca 3, HR-10000 Zagreb, Croatia † Center for Language Technology, University of Copenhagen Njalsgade 140, 2300 Copenhagen S, Denmark ‡ Institute of Croatian Language and Linguistics Ulica Republike Austrije 16, HR-10000 Zagreb, Croatia {nljubesi,fklubick}@ffzg.hr zeljko.agic@hum.ku.dk ipjazbec@ihjj.hr Abstract In this paper we present newly developed inflectional lexcions and manually annotated corpora of Croatian and Serbian. We introduce hrLex and srLex—two freely available inflectional lexicons of Croatian and Serbian—and describe the process of building these lexicons, supported by supervised machine learning techniques for lemma and paradigm prediction. Furthermore, we introduce hr500k, a manually annotated corpus of Croatian, 500 thousand tokens in size. We showcase the three newly developed resources on the task of morphosyntactic annotation of both languages by using a recently developed CRF tagger. We achieve best results yet reported on the task for both languages, beating the HunPos baseline trained on the same datasets by a wide margin. Keywords: inflectional lexicon, morphosyntactic annotation, Croatian, Serbian 1. Introduction In this paper we introduce hrLex and srLex, freely available morphological lexicons of Croatian and Serbian. We de- scribe the process of building these lexicons which includes supervised machine learning techniques for lemma and paradigm candidate ranking of non-covered words to en- hance the linguists’ productivity. Furthermore, we present hr500k, a new Croatian gold dataset manually annotated with morphosyntactic and lemma information, 500 thou- sand tokens in size. The dataset represents a balanced ex- tension of the SETimes.HR dataset (Agi´c and Ljubeˇsi´c, 2014) which consisted of newspaper articles from one source only. We perform an intrinsic evaluation of the inflectional lex- icons and an extrinsic evaluation of all three resources on the task of morphosyntactic tagging of both Croatian and Serbian. We compare the newly developed corpus with its predecessor, the SETimes.HR corpus, and measure the accuracy gain obtained by including the lexicon in the morphosyntactic tagging process. For extrinsic evaluation we use a recently developed tagger optimised on Slovene (Ljubeˇsi´c and Erjavec, 2016), comparing its results to the ones obtained with the popular HunPos tagger (Hal´acsy et al., 2007) when trained on the same datasets. 2. Related Work For both languages morphological lexicons were developed in the past, but with limited availability. For Croatian the Croatian Morphological Lexicon (Tadi´c and Fulgosi, 2003) was available for search through a web interface since 2005 (Tadi´c, 2005). Since 2012 this lexicon is available through Meta-Share, with a size of ca 113,000 lemmas (60% of which are proper names) in version 5.0. However, it is dis- tributed under a non-commercial license in the form of (to- ken, lemma, tag) triples only, and is therefore not useful for expansion or enrichment. ˇSnajder et al. (2008) provide an- other line of work on Croatian inflectional lexica, but the resulting resource is not freely available. For Serbian the SrpMD dictionary (Krstev, 2008), 85,721 lemmas in size, …

Content truncated to 500 words