Assembling the Kazakh Language Corpus

Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1022–1031, Seattle, Washington, USA, 18-21 October 2013. c⃝2013 Association for Computational Linguistics Assembling the Kazakh Language Corpus Olzhas Makhambetov, Aibek Makazhanov, Zhandos Yessenbayev, Bakhyt Matkarimov, Islam Sabyrgaliyev, and Anuar Sharafudinov Nazarbayev University Research and Innovation System 53 Kabanbay batyr ave., Astana, Kazakhstan {omakhambetov, aibek.makazhanov, zhyessenbayev, bmatkarimov, islam.sabyrgaliyev, anuar.sharaphudinov}@nu.edu.kz Abstract This paper presents the Kazakh Language Corpus (KLC), which is one of the first attempts made within a local research community to assemble a Kazakh corpus. KLC is designed to be a large scale corpus containing over 135 million words and con- veying five stylistic genres: literary, publicistic, of- ficial, scientific and informal. Along with its pri- mary part KLC comprises such parts as: (i) anno- tated sub-corpus, containing segmented documents encoded in the eXtensible Markup Language (XML) that marks complete morphological, syntactic, and structural characteristics of texts; (ii) as well as a sub-corpus with the annotated speech data. KLC has a web-based corpus management system that helps to navigate the data and retrieve necessary informa- tion. KLC is also open for contributors, who are willing to make suggestions, donate texts and help with annotation of existing materials. 1 Introduction This article describes theoretical and practical issues ex- perienced during the construction of the Kazakh Lan- guage Corpus. Kazakh is an agglutinative and highly in- flected language which belongs to the Turkic group. It is official state language of Kazakhstan and a mother tongue for more than 10 million people all around the world. However, up until the early 90’s of 20th century, due to historical reasons of the Soviet era, Russian language was the predominant language in spoken and written commu- nication in Kazakhstan. This fact in turn caused the prob- lem of underrepresentation of Kazakh language in var- ious fields such as science, entertainment, official doc- umentation, etc. For this reason, while assembling the corpus, we had to group categories that are generally pre- sented as separate in other corpora into five stylistic gen- res. Also, in contrast to other corpora (Aksan et al., 2012; Chen, 1996), we included texts as they were available, i.e we did not try to fill a predefined set of categories. Substantial part of materials was collected using source- customized web crawlers and donated texts. KLC also contains a manually annotated sub-corpus with morpho-syntactic and structural markups encoded in XML following general notions outlined in CES (Ide, 1998). Our syntactic tagset comprises a set of syntactic categories well-defined in a classical Kazakh grammar, and the part of speech (POS) tagset is based on a posi- tional system in which the tags are formed by concatena- tion of POS of a word form and a chain of encoded lin- guistic properties, such as number, case, voice etc. The annotations have been carried out manually by philology students specializing in morphology and syntax. Trying to make the annotation process as comfortable as possi- ble, we have designed a web-based annotation tool with a …

Content truncated to 500 words