UniMorph 2.0: Universal Morphology

UniMorph 2.0: Universal Morphology Christo Kirov1, Ryan Cotterell1, John Sylak-Glassman1, G´eraldine Walther2 Ekaterina Vylomova3, Patrick Xia1, Manaal Faruqui4, Sabrina J. Mielke1, Arya D. McCarthy1 Sandra K¨ubler5, David Yarowsky1, Jason Eisner1, Mans Hulden6 1Johns Hopkins University, 2University of Zurich, 3University of Melbourne 4Google, 5Indiana University, 6University of Colorado Baltimore, Zurich, Melbourne, New York, Bloomington, Boulder {ckirov1, ryan.cotterell, jcsg, paxia, sjmielke, arya, yarowsky, eisner}@jhu.edu, geraldine.walther@uzh.ch evylomova@gmail.com, mfaruqui@google.com, skuebler@indiana.edu, mans.hulden@colorado.edu Abstract The Universal Morphology (UniMorph) project is a collaborative effort to improve how NLP handles complex morphology across the world’s languages. The project releases annotated morphological data using a universal tagset, the UniMorph schema. Each inflected form is associated with a lemma, which typically carries its underlying lexical meaning, and a bundle of morphological features from our schema. Additional supporting data and tools are also released on a per-language basis when available. UniMorph is based at the Center for Language and Speech Processing (CLSP) at Johns Hopkins University in Baltimore, Maryland. This paper details advances made to the collection, annotation, and dissemination of project resources since the initial UniMorph release described at LREC 2016. Keywords: morphology, multilingual, lexical resources 1. Introduction Complex morphology is ubiquitous among the languages of the world. For example, roughly 80% of languages use morphology to mark verbal tense and 65% mark nomi- nal case (Haspelmath et al., 2005). While overlooked in the past, explicit modeling of morphology has been shown to improve performance on a number of downstream HLT tasks, including including machine translation (MT) (Dyer et al., 2008), speech recognition (Creutz et al., 2007), parsing (Seeker and C¸ etinoˇglu, 2015), keyword spotting (Narasimhan et al., 2014), and word embedding (Cotterell et al., 2016b). This has led to a surge of new interest and work in this area (Durrett and DeNero, 2013; Ahlberg et al., 2014; Nicolai et al., 2015; Faruqui et al., 2016). The Universal Morphology (UniMorph) project, centered at the Center for Language and Speech Processing (CLSP) at Johns Hopkins University is a collaborative effort to improve how NLP systems handle complex morphology across the world’s languages. The project releases anno- tated morphological data using a universal tagset, the Uni- Morph schema. Each inflected form is associated with a lemma, which typically carries its underlying lexical meaning, and a bundle of morphological features from our schema. Additional supporting data and tools are also re- leased on a per-language basis when available. Kirov et al. (2016) introduced version 1.0 of the UniMorph morphological database, created by extracting and normal- izing the inflectional paradigms included in Wiktionary (www.wiktionary.org), a large, broadly multi-lingual crowd-sourced collection of lexical data. This paper de- scribes UniMorph 2.0. It details improvements in Wik- tionary extraction and annotation, as well as normaliza- tion of non-Wiktionary resources, leading to a much higher quality morphological database. The new dataset spans 52 languages representing a range of language families. As in UniMorph 1.0, we provide paradigms from highly- inflected open-class word categories — nouns, verbs, and adjectives. Many of the included languages are extremely low-resource, e.g., Quechua, …

Content truncated to 500 words