UDPipe: Trainable Pipeline for Processing CoNLL-U Files Performing Tokenization, Morphological Analysis, POS Tagging and Parsing

UDPipe: Trainable Pipeline for Processing CoNLL-U Files Performing Tokenization, Morphological Analysis, POS Tagging and Parsing Milan Straka, Jan Hajiˇc, Jana Strakov´a Charles University in Prague Faculty of Mathematics and Physics Institute of Formal and Applied Linguistics {straka,hajic,strakova}@ufal.mff.cuni.cz Abstract Automatic natural language processing of large texts often presents recurring challenges in multiple languages: even for most advanced tasks, the texts are first processed by basic processing steps – from tokenization to parsing. We present an extremely simple-to-use tool consisting of one binary and one model (per language), which performs these tasks for multiple languages without the need for any other external data. UDPipe, a pipeline processing CoNLL-U-formatted files, performs tokenization, morphological analysis, part-of-speech tagging, lemmatization and dependency parsing for nearly all treebanks of Universal Dependencies 1.2 (namely, the whole pipeline is currently available for 32 out of 37 treebanks). In addition, the pipeline is easily trainable with training data in CoNLL-U format (and in some cases also with additional raw corpora) and requires minimal linguistic knowledge on the users’ part. The training code is also released. Keywords: Universal Dependencies; dependency parsing; part-of-speech tagging 1. Introduction The Universal Dependencies project (Nivre et al., 2016) seeks to develop cross-linguistically consistent treebank an- notation for many languages. The annotation scheme is based on the universal Stanford dependencies (de Marn- effe et al., 2014), the Google universal part-of-speech tags (Petrov et al., 2012), and the Interset interlingua for mor- phosyntactic features (Zeman, 2008). The latest version of UD (Univeral Dependencies Treebanks version 1.2, 2015)1 consists of 37 dependency treebanks. Our aim is to create a simple-to-use, one-binary and a single-model tool (per language), to easily process raw text to CoNLL-U-formatted tagged and/or parsed dependency trees (with morphological features if available in UD). In particular, our goals are • state-of-the-art tools for tokenization, morphological analysis, part-of-speech tagging and dependency pars- ing, • free C++ tools available under the Mozilla Public Li- cense (MPL) 2.0 license (code) and CC BY-NC-SA 4.0 license (models), • simple tool with a single model (per language), easily trainable with custom data (CoNLL-U format), • trained models provided for as many UD treebanks as possible, • no feature engineering, no external morphological dic- tionary, no language specific knowledge, • efficient programming design in terms of RAM and disc usage. Following these requirements, we developed a morpholog- ical dictionary tool and tagger software, which is described in Section 4. and in detail in (Strakov´a et al., 2014), and a dependency parser described in Section 5. and in detail in (Straka et al., 2015). We wrapped these tools along with a trainable tokenizer based on artificial neural networks (Sec- 1http://hdl.handle.net/11234/1-1548 tion 3., which we describe in a little more detail since it has not been published previously) in a single tool called UD- Pipe, described in Section 6. We also compare our set of tools contained in the UDPipe with other results, expecially other dependency parsers. 2. Related Work A considerable number of natural language processing pipelines are available, e.g. OpenNLP2 or Natural Lan- …

Content truncated to 500 words