IceBATS: An Icelandic Adaptation of the Bigger Analogy Test Set

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 417–426 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 417 KazNERD: Kazakh Named Entity Recognition Dataset Rustem Yeshpanov, Yerbolat Khassanov, Huseyin Atakan Varol Institute of Smart Systems and Artificial Intelligence, Nazarbayev University, Nur-Sultan, Kazakhstan {rustem.yeshpanov, yerbolat.khassanov, ahvarol}@nu.edu.kz Abstract We present the development of a dataset for Kazakh named entity recognition. The dataset was built as there is a clear need for publicly available annotated corpora in Kazakh, as well as annotation guidelines containing straightforward—but rigorous—rules and examples. The dataset annotation, based on the IOB2 scheme, was carried out on television news text by two native Kazakh speakers under the supervision of the first author. The resulting dataset contains 112,702 sentences and 136,333 annotations for 25 entity classes. State-of-the-art machine learning models to automatise Kazakh named entity recognition were also built, with the best-performing model achieving an exact match F1-score of 97.22% on the test set. The annotated dataset, guidelines, and codes used to train the models are freely available for download under the CC BY 4.0 licence from https://github.com/IS2AI/KazNERD. Keywords: Named Entity Recognition, NER, Kazakh, Dataset, Annotation Guidelines, CRF, BiLSTM, BERT 1. Introduction Named entity recognition (NER) refers to a subtask of information extraction aimed at identifying named entities (NEs) in semi- or unstructured text and classifying them into pre-specified types (Nadeau and Sekine, 2007). NEs, in turn, generally refer to (proper) names of persons, organisations, and geographical locations (Sang and Meulder, 2003), as well as numerical and temporal expressions, including quantities, monetary units, percentages, dates, or durations (Chinchor, 1998). Widely used in natural language processing applications, including automatic text understanding (Cheng and Erk, 2020), machine translation (Babych and Hartley, 2003), question answering (Aliod et al., 2006), and knowledge base development (Etzioni et al., 2005) to name a few, NER has been of interest not only to scientific research, but also to business (Sch¨on et al., 2019) and defence (Han et al., 2020) ever since 1995, when the term was coined (Grishman and Sundheim, 1996). By virtue of most of the early works in information extraction being launched as part of United States Government initiatives (e.g., ACE, MUC, TIPSTER) (Maynard et al., 2003), a great deal of research in NER concerns English. Nonetheless, an equally large proportion of NER research has been dedicated to different well- resourced languages, such as Spanish, French, German, Japanese, Chinese, Russian (see Nadeau and Sekine (2007), for a detailed overview), as well as to less resourced ones, such as Sindhi (Ali et al., 2020), Romanian (Dumitrescu and Avram, 2020), and Icelandic (Ing´olfsd´ottir et al., 2019). Likewise low-resourced, the language of interest of this paper—Kazakh—has only latterly appeared on the radar of NER researchers. Underrepresented and lexically underdeveloped because overshadowed by Russian, which was promoted as a lingua franca during the Soviet era (Dave, 2007), the earliest NER research in this agglutinative Turkic language dates back as recently as 2016. Although there is evidence for annotated corpus construction …

Content truncated to 500 words