Very-large Scale Parsing and Normalization of Wiktionary Morphological Paradigms

Very-large Scale Parsing and Normalization of Wiktionary Morphological Paradigms Christo Kirov, John Sylak-Glassman, Roger Que, David Yarowsky Center for Language and Speech Processing, Johns Hopkins University Baltimore, MD 21218 ckirov@gmail.com, jcsg@jhu.edu, query@jhu.edu, yarowsky@jhu.edu Abstract Wiktionary is a large-scale resource for cross-lingual lexical information with great potential utility for machine translation (MT) and many other NLP tasks, especially automatic morphological analysis and generation. However, it is designed primarily for human viewing rather than machine readability, and presents numerous challenges for generalized parsing and extraction due to a lack of standardized formatting and grammatical descriptor definitions. This paper describes a large-scale effort to automatically extract and standardize the data in Wiktionary and make it available for use by the NLP research community. The methodological innovations include a multidimensional table parsing algorithm, a cross-lexeme, token-frequency-based method of separating inflectional form data from grammatical descriptors, the normalization of grammatical descriptors to a unified annotation scheme that accounts for cross-linguistic diversity, and a verification and correction process that exploits within-language, cross-lexeme table format consistency to minimize human effort. The effort described here resulted in the extraction of a uniquely large normalized resource of nearly 1,000,000 inflectional paradigms across 350 languages. Evaluation shows that even though the data is extracted using a language-independent approach, it is comparable in quantity and quality to data extracted using hand-tuned, language-specific approaches. Keywords: Wiktionary, Morphology, UniMorph, Multilingual Resources 1. Introduction Wiktionary1 is one of the largest sources of lexemes (or lemmas) with morphological paradigms across a wide range of languages, and exhibits substantial ongoing growth. This makes it an ideal source of data for broadly cross-lingual machine learning in NLP, with potential ap- plications in machine translation, information extraction, and many other tasks. However, the human-focused, crowd-sourced nature of Wiktionary presents a number of challenges to gathering and using this data. Wiktionary is composed of a number of editions, with each edition written for readers of a particular language (e.g., the French edition is written for French readers). Each edition covers lemmas in many languages (i.e., both the French and English editions of Wiktionary contain Swahili lemmas). There is no fixed set of standards for how lexical informa- tion should be presented. This leads to many formatting idiosyncrasies across editions, languages, and lemmas. Our aim is to find and exploit any regularities in how authors choose to represent lemmas and morphological paradigms to extract and standardize the information in Wiktionary into a form suitable for machine processing. Our current focus for this effort is capturing inflectional morphology, although we also extract data relevant to derivational morphology and word-to-word translation. To achieve standardized output, we develop an original, ro- bust parsing algorithm that operates directly on the sur- face HTML of Wiktionary, and requires minimal edition- specific or language-specific tuning. The output of our gen- eralized approach produces data of similar quality to pre- vious fine-tuned language-specific extractors, although in much greater quantities. Our code and data will be made available under an open-source license as part of the Uni- 1http://www.wiktionary.org …

Content truncated to 500 words