Wiktextract: Wiktionary as Machine-Readable Structured Data

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 1317–1325 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 1317 Wiktextract: Wiktionary as Machine-Readable Structured Data Tatu Ylonen University of Helsinki Department of Digital Humanities / Language Technology tatu@ylonen.org Abstract We present a machine-readable structured data version of Wiktionary. Unlike previous Wiktionary extractions, the new extractor, Wiktextract, fully interprets and expands templates and Lua modules in Wiktionary. This enables it to perform a more complete, robust, and maintainable extraction. The extracted data is multilingual and includes lemmas, inflected forms, translations, etymology, usage examples, pronunciations (including URLs of sound files), lexical and semantic relations, and various morphological, syntactic, semantic, topical, and dialectal annotations. We extract all data from the English Wiktionary. Comparing against previous extractions from language-specific dictionaries, we find that its coverage for non-English languages often matches or exceeds the coverage in the language-specific editions, with the added benefit that all glosses are in English. The data is freely available and regularly updated, enabling anyone to add more data and correct errors by editing Wiktionary. The extracted data is in JSON format and designed to be easy to use by researchers, downstream resources, and application developers. Keywords: Wiktionary, extraction, dictionary, lexicon, andmorphology, inflections, translation, etymology, dialects 1. Introduction Wiktionary1 is undoubtedly the world’s largest freely available dictionary. While many partial extractors ex- ist, no-one seems to have succeeded in fully convert- ing Wiktionary into a computationally easy-to-use ma- chine readable format. This is largely because it is an unstructured wiki maintained by thousands of volun- teers with varying technical skills. Tens of thousands of templates and programmed extension modules in the Lua programming language2 are used by the volunteers for formatting content. It is these templates and Lua modules, combined with the great variety of ways the information is encoded by the volunteers for hundreds of languages that have made general extraction so dif- ficult. Lua modules are used to generate pronuncia- tions, inflection tables, and even entire sets of glosses for many languages. Over the last several years, we have developed new Python packages for computationally processing Wik- tionary. The wikitextprocessor3 package im- plements parsing wikitext4 (the source format used for all Wiktionary and Wikipedia language variants), ex- panding templates, and executing Lua modules. The wiktextract5 tool implements converting the En- glish Wiktionary into a machine-readable, structured format. The English Wiktionary has glosses in English, but it contains words and translations for hundreds of different languages. We have furthermore implemented other tools that present the extracted data as a web site6 for testing, 1https://wiktionary.org 2https://www.lua.org 3https://github.com/tatuylonen/wikitextprocessor 4https://en.wikipedia.org/wiki/Help:Wikitext 5https://github.com/tatuylonen/wiktextract 6https://kaikki.org evaluation, post-processing, and for making the data available to others. The result of this work is a machine-readable version of Wiktionary, formatted as JSON7 to make it as easy as possible for others to use. We regularly update the extraction, so that any words added to Wiktionary or improvements made to existing words will soon appear in the extraction. The extraction includes …

Content truncated to 500 words