EstNLTK 1.6: Remastered Estonian NLP Pipeline

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 7152–7160 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 7152 EstNLTK 1.6: Remastered Estonian NLP Pipeline Sven Laur, Siim Orasmaa, Dage S¨arg, Paul Tammo Institute of Computer Science, University of Tartu Liivi 2, 50409 Tartu, Estonia {sven.laur, siim.orasmaa, dage.sarg}@ut.ee, paul.tammo@gmail.com Abstract The goal of the ESTNLTK Python library is to provide a unified programming interface for natural language processing in Estonian. As such, previous versions of the library have been immensely successful both in academic and industrial circles. However, they also contained serious structural limitations – it was hard to add new components and there was a lack of fine-grained control needed for back-end programming. These issues have been explicitly addressed in the ESTNLTK library while preserving the intuitive interface for novices. We have remastered the basic NLP pipeline by adding many data cleaning steps that are necessary for analyzing real-life texts, and state of the art components for morphological analysis and fact extraction. Our evaluation on unlabelled data shows that the remastered basic NLP pipeline outperforms both the previous version of the toolkit, as well as neural models of StanfordNLP. In addition, ESTNLTK contains a new interface for storing, processing and querying text objects in POSTGRES database which greatly simplifies processing of large text collections. ESTNLTK is freely available under the GNU GPL version 2 license, which is standard for academic software. Keywords: natural language processing, Python, Estonian language 1. Introduction Estonian natural language processing has seen steady ad- vancements in recent years. As a part of the Universal Dependencies effort (Nivre et al., 2016), Estonian tree- banks with standardized grammatical annotations have be- come available. The Universal Dependencies effort has fostered the experimentation with multilingual data-driven parsing models, many of which also cover Estonian (Ze- man et al., 2018; Yusupujiang, 2018). In the wake of these developments, a new paradigm – data-driven neu- ral models – has been introduced into Estonian morpho- logical analysis (Tkachenko and Sirts, 2018). Still, rule- based linguistic analysis models are also being successfully implemented. Kaalep et al. (2018) introduced a finite- state-transducers-based morphological analyzer for Esto- nian which uses a revised morphological category system and provides a more detailed analysis for inflectional forms and compound words. Universal Dependencies data sets have enabled the inclu- sion of automatic analysis of Estonian in multilingual NLP tools, such as StanfordNLP (Qi et al., 2018) and NLP-Cube (Boros¸ et al., 2018). Harnessing the power of deep neural networks, these tools provide end-to-end text analysis ca- pabilities, starting from text segmentation into words and sentences, and leading up to morphological analysis and syntactic parsing. While these pipelines provide basic anal- ysis for standard language, they offer no convenient ways for adapting to specific text domains – apart from amass- ing a large amount of training data and retraining the sys- tem. However, when dealing with real-life text analysis problems, cost-effective ways for pipeline adaptation are often needed. For instance, in …

Content truncated to 500 words