Extracting Weighted Language Lexicons from Wikipedia

Extracting Weighted Language Lexicons from Wikipedia Gregory Grefenstette Inria Saclay/TAO, Rue Noetzlin - Bât 660 91190 Gif sur Yvette, France E-mail: gregory.grefenstette@inria.fr Abstract Language models are used in applications as diverse as speech recognition, optical character recognition and information retrieval. They are used to predict word appearance, and to weight the importance of words in these applications. One basic element of language models is the list of words in a language. Another is the unigram frequency of each word. But this basic information is not available for most languages in the world. Since the multilingual Wikipedia project encourages the production of encyclopedic-like articles in many world languages, we can find there an ever-growing source of text from which to extract these two language modelling elements: word list and frequency. Here we present a simple technique for converting this Wikipedia text into lexicons of weighted unigrams for the more than 270 languages present currently present in Wikipedia. The lexicons produced, and the source code for producing them in a Linux-based system are here made available for free on the Web. Keywords: lexicon, Wikipedia, language model, under-resourced languages 1. Introduction Most natural language processing applications involve language-specific processing tools, and one essential element in these tools is a lexicon of words for the language processed. In the best cases these lexicons contain morphological, grammatical and usage information for each word (Sérasset, 2012) (Hathout et al., 2014). In the worst cases, which is the case for the majority of the 6000+ languages spoken in the world, no computational lexicons are available1. Thus, a useful first step for under-resourced would be to provide at least a list of known words in that language. In addition to having a word list for the language, providing unigram probabilities, or relative frequencies of the known words, allows some natural language applications, such as optical character reading, spelling correction (Li, et al., 2011) machine translation (Tiers & Pienaar, 2008), or information retrieval, to make better decisions. There are a few automatic ways to produce lexicons for languages. One way is to crawl the web and using a language identifier to classify pages into languages. Then, once boilerplate is removed, one can extract a lexicon from these pages (See, for example Erjavec et al., 2008). In order to start a targeted crawling (Medelyan et al., 2006) , one starts from a list of URLs known to be in the desired language and a list of words to filter new pages discovered during the crawl. The Crúbadán Project (Scannell, 2007) uses such a crawler to continuously gather text for under-resourced languages. So far, the Crúbadán project has created lexicons for 2124 writing systems. For each language, anyone can download2 a list of URLs that have been gleaned and hand edited for the language, and a list of character tri-grams that can be used to create a language identifier (Grefenstette, 1995). The Crúbadán pages also make available a weighted lexicon, such as we present here, 1 https://www.ethnologue.com/world 2 http://crubadan.org/writingsystems from this …

Content truncated to 500 words