A New Integrated Open-source Morphological Analyzer for Hungarian

A New Integrated Open-source Morphological Analyzer for Hungarian Attila Nov´ak1,2, Borb´ala Sikl´osi2, Csaba Oravecz3 1MTA-PPKE Hungarian Language Technology Research Group Pr´ater u. 50/a, 1083 Budapest, Hungary 2 P´azm´any P´eter Catholic University, Faculty of Information Technology and Bionics Pr´ater u. 50/a, 1083 Budapest, Hungary 3Research Institute for Linguistics, Hungarian Academy of Sciences Bencz´ur u. 33, 1068 Budapest, Hungary {novak.attila, siklosi.borbala}@itk.ppke.hu, oravecz.csaba@nytud.mta.hu Abstract The goal of a Hungarian research project has been to create an integrated Hungarian natural language processing framework. This infrastructure includes tools for analyzing Hungarian texts, integrated into a standardized environment. The morphological analyzer is one of the core components of the framework. The goal of this paper is to describe a fast and customizable morphological analyzer and its development framework, which synthesizes and further enriches the morphological knowledge implemented in previous tools existing for Hungarian. In addition, we present the method we applied to add semantic knowledge to the lexical database of the morphology. The method utilizes neural word embedding models and morphological and shallow syntactic knowledge. Keywords: morphological analysis, agglutinating languages, word embedding 1. Introduction With a joint effort of leading Hungarian research centers1 in NLP, a project has been launched to develop a unified, in- tegrated framework consisting of a basic language process- ing toolkit (BLARK) for the language. All of the compo- nents are designed to synthesize previous development ef- forts and to deliver the best quality and performance avail- able, with a focus not only on language technology but also on serving the purposes of (theoretical) linguistic research. The infrastructure includes tools and resources from speech technology to syntactic analysis and is integrated into a standardized environment. All tools will be licensed open source, free to use for research and commercial purposes and will be available as plugins in the GATE architecture (Gaizauskas et al., 1996). The resulting framework be- ing part of the national strategic research infrastructure will stand as the de facto reference NLP toolkit for the Hungar- ian language. The infrastructure will implement a complete NLP pipeline including a tokenizer, a morphological analyzer, a part of speech tagger and a constituent and dependency parser. As auxiliary tools, a shallow parser (NP chunker) and a named entity recognizer will also be incorporated into the toolkit. For highly inflectional languages like Finnish or Hungar- ian, a morphological analyzer is one of the core compo- nents of an NLP tool chain and its precision and coverage are critical for higher-level processing tasks. The goal of this paper is twofold: (i) to report on the development of a fast and customizable morphological analyzer, which syn- thesizes and further enriches the morphological knowledge implemented in previous tools existing for Hungarian; (ii) to present an algorithm based on neural word embedding 1Research Institute for Linguistics, Hungarian Academy of Sciences; MTA-PPKE Hungarian Language Technology Research Group, P´azm´any P´eter Catholic University; Research Group on Artificial Intelligence, University of Szeged models and morphological and shallow syntactic knowl- edge to add semantic knowledge to the lexical database of the morphology. 2. …

Content truncated to 500 words