E-magyar – A Digital Language Processing System

e-magyar – A Digital Language Processing System Tam´as V´aradi1, Eszter Simon1, B´alint Sass1, Iv´an Mittelholcz1, Attila Nov´ak2,3, Bal´azs Indig2,3, Rich´ard Farkas4, Veronika Vincze4,5 1Research Institute for Linguistics, Hungarian Academy of Sciences H-1068 Budapest, Bencz´ur u. 33. 2MTA-PPKE Hungarian Language Technology Research Group 3P´azm´any P´eter Catholic University, Faculty of Information Technology and Bionics H-1083 Budapest, Pr´ater u. 50/a 4University of Szeged, Institute of Informatics H-6720 Szeged, ´Arp´ad t´er 2. 5MTA-SZTE Research Group on Artificial Intelligence H-6720 Szeged, Tisza Lajos krt. 103. {varadi.tamas,simon.eszter,sass.balint,mittelholcz.ivan}@nytud.mta.hu, {novak.attila,indig.balazs}@itk.ppke.hu, {rfarkas,vinczev}@inf.u-szeged.hu Abstract e-magyar is a new toolset for the analysis of Hungarian texts. It was produced as a collaborative effort of the Hungarian language technology community integrating the best state-of-the-art tools, enhancing them where necessary, making them interoperable and releasing them with a clear license. It is a free, open, modular text processing pipeline which is integrated in the GATE system offering further prospects of interoperability. From tokenizing to parsing and named entity recognition, existing tools were examined and those selected for integration underwent various amount of overhaul in order to operate in the pipeline with a uniform encoding, and run in the same Java platform. The tokenizer was re-built from ground up and the flagship module, the morphological analyzer, based on the Humor system (Pr´osz´eky and Kis, 1999), was given a new annotation system and was implemented in the HFST framework (Lind´en et al., 2009). The system is aimed for a broad range of users, from language technology application developers to digital humanities researchers alike. It comes with a drag-and-drop demo on its website: http://e-magyar.hu/en/. Keywords: text analysis, Hungarian pipeline, integrated toolset 1. Introduction The paper describes e-magyar, a new integrated text pro- cessing pipeline for Hungarian. While Hungarian can be considered an under-resourced language it does have an active and cooperating language technology community which has been developing various tools to cover the ba- sic text processing steps. However, earlier fragmented ef- forts suffered from a number of factors such as the lack of interoperability, openness, clearly defined license con- ditions and/or have become limited in some technological aspects such as encoding and annotation systems used as well as efficiency and implementation platform. All these reasons served as the motivation for a collaborative effort by key Hungarian language technology partners to overhaul (sometimes quite radically) the existing tools and, more im- portantly, to make them interoperable so that they can be integrated into a single coherent technological pipeline. The objectives of the e-magyar system were to serve as an open, free and modular text processing toolset that serves the needs of commercial developers and individual researchers in language technology as well as (digital) hu- manities and social sciences. It is open and free in that the system as a whole and its individual components are typ- ically available for download through Github repositories, freely available often not just for research and development but for commercial use and they certainly come with clear license terms. Technologically, the aim was to take the state-of-the-art tools available, eliminate their …

Content truncated to 500 words