Development and Evaluation of Three Named Entity Recognition Systems for Serbian - The Case of Personal Names

Proceedings of Recent Advances in Natural Language Processing, pages 1060–1068, Varna, Bulgaria, Sep 2–4, 2019. https://doi.org/10.26615/978-954-452-056-4_122 1060 Development and Evaluation of Three Named Entity Recognition Systems for Serbian - The Case of Personal Names Branislava Šandrih University of Belgrade Faculty of Philology Belgrade, Serbia branislava.sandrih@fil.bg.ac.rs Cvetana Krstev University of Belgrade Faculty of Philology Belgrade, Serbia cvetana@matf.bg.ac.rs Ranka Stankovi´c University of Belgrade Faculty of Mining and Geology Belgrade, Serbia ranka@rgf.bg.ac.rs Abstract In this paper we present a rule- and lexicon-based system for the recognition of Named Entities (NE) in Serbian news- paper texts that was used to prepare a gold standard annotated with personal names. It was further used to prepare training sets for four different levels of annota- tion, which were further used to train two Named Entity Recognition (NER) sys- tems: Stanford and spaCy. All obtained models, together with a rule- and lexicon- based system were evaluated on two sam- ple texts: a part of the gold standard and an independent newspaper text of approx- imately the same size. The results show that rule- and lexicon-based system out- performs trained models in all four sce- narios (measured by F1), while Stanford models have the highest recall. The pro- duced models are incorporated into a Web platform NER&Beyond that provides vari- ous NE-related functions. 1 Introduction Named Entity Recognition is the task of identi- fying named entities in text (Nadeau and Sekine, 2007), which is often used as a first step in ques- tion answering, information retrieval, anaphora resolution, topic modeling, etc. The first Named Entity set had 7 types (Grishman and Sundheim, 1996): organization, location, person, date, time, money and percent expressions. Sekine et al. (2002) proposed a NE hierarchy which contains about 150 NE types. There are three categories of NER systems: 1) The rule-based (RB) (Krupka and Hausman, 1998; Friburger and Maurel, 2004); 2) the Ma- chine Learning (ML) based (Finkel and Manning, 2009; Singh et al., 2010); and 3) hybrid meth- ods (Jansche and Abney, 2002). The ML-based methods can often be “black boxes”, in compar- ison with RB techniques which are easy to in- terpret. Yet, ML-based methods are state-of-the- art. Such example is a Stanford Named Entity Recognizer (Manning et al., 2014), which can be trained for many languages. Other notable NER platforms include GATE (Desktop appli- cation that enables NER across many languages and domains),1 OpenNLP (rule-based and statisti- cal NER),2 spaCy (Honnibal and Montani, 2017) (module written in Python, used for advanced NLP)3 and many others. For Serbian, thus far a rule-based and lexicon-based NER system was developed – SRPNER (Krstev et al., 2014). Its development started with the recognition of a NE class present in all NE schemes, personal names (Krstev et al., 2005), while the recognition of other main NE classes was subsequently added. In the next Sec- tion we present briefly this system and how it was used to produce the corpus of newspaper texts an- notated with personal names – the gold standard. Section 3 describes NER systems …

Content truncated to 500 words