An Open, Extendible, and Fast Turkish Morphological Analyzer

Proceedings of Recent Advances in Natural Language Processing, pages 1364–1372, Varna, Bulgaria, Sep 2–4, 2019. https://doi.org/10.26615/978-954-452-056-4_156 1364 An Open, Extendible, and Fast Turkish Morphological Analyzer Olcay Taner Yıldız1, Beg¨um Avar2, G¨okhan Ercan1 1 Department of Computer Engineering, Is¸ık University, ˙Istanbul, Turkey 2 Department of Linguistics, Bo˘gazic¸i University, ˙Istanbul, Turkey olcaytaner@isikun.edu.tr, begum.avar@boun.edu.tr gokhan.ercan@isik.edu.tr Abstract In this paper, we present a two-level morphological analyzer for Turkish which consists of five main components: finite state transducer, rule engine for suffixa- tion, lexicon, trie data structure, and LRU cache. We use Java language to imple- ment finite state machine logic and rule engine, Xml language to describe the fi- nite state transducer rules of the Turkish language, which makes the morphological analyzer both easily extendible and eas- ily applicable to other languages. Em- powered with a comprehensive lexicon of 54,000 bare-forms including 19,000 proper nouns, our morphological analyzer is amongst the most reliable analyzers pro- duced so far. The analyzer is compared with Turkish morphological analyzers in the literature. By using LRU cache and a trie data structure, the system can ana- lyze 100,000 words per second, which en- ables users to analyze huge corpora in a few hours. 1 Introduction Morphological analysis is one of the key compo- nents of computational language processing, es- pecially in morphologically rich languages. Af- ter some preprocessing stages such as stripping nontextual parts from the corpus and sentence seg- mentation, the first and foremost stage in computa- tional analysis of the text is morphological analy- sis, that is dividing the words into constituent mor- phemes. In this study, we deal with the morphology of Turkish, which is a textbook example for an ag- glutinative language. In Turkish, a word in its sur- face form contains 3 to 4 morphemes on the aver- age (Sak, 2011), and these morphemes can have a semantic and/or syntactic content. Not only a sur- face form can have a multimorpheme structure in Turkish, but also has multi morphological analy- ses. In this paper, we will present a new morpholog- ical analyzer, which is (i) open: The latest version of source codes, the lexicon, and the morphotac- tic rule engine are all available on the Internet1, (ii) extendible: One of the disadvantages of other morphological analyzers is that their lexicons are fixed or unmodifiable, which prevents to add new bare-forms to the morphological analyzer. In our morphological analyzer, the lexicon is in text form and is easily modifiable, (iii) fast: Morphological analysis is one of the core components of any NLP process. It must be very fast to handle huge cor- pora. Compared to other morphological analyz- ers, our analyzer is capable of analyzing hundreds of thousands words per second, which makes it one of the fastest Turkish morphological analyz- ers available. 2 Turkish Morphology In linguistics, the term morphology refers to the study of the internal structure of words. Each word is assumed to consist of one or more morphemes, which can be defined as the smallest linguistic unit having a particular …

Content truncated to 500 words