WIKIPARQ: A Tabulated Wikipedia Resource Using the Parquet Format

WikiParq: A Tabulated Wikipedia Resource Using the Parquet Format Marcus Klang, Pierre Nugues Lund University, Department of Computer science, Lund, Sweden marcus.klang@cs.lth.se, pierre.nugues@cs.lth.se Abstract Wikipedia has become one of the most popular resources in natural language processing and it is used in quantities of applications. However, Wikipedia requires a substantial pre-processing step before it can be used. For instance, its set of nonstandardized annotations, referred to as the wiki markup, is language-dependent and needs specific parsers from language to language, for English, French, Italian, etc. In addition, the intricacies of the different Wikipedia resources: main article text, categories, wikidata, infoboxes, scattered into the article document or in different files make it difficult to have global view of this outstanding resource. In this paper, we describe WikiParq, a unified format based on the Parquet standard to tabulate and package the Wikipedia corpora. In combination with Spark, a map-reduce computing framework, and the SQL query language, WikiParq makes it much easier to write database queries to extract specific information or subcorpora from Wikipedia, such as all the first paragraphs of the articles in French, or all the articles on persons in Spanish, or all the articles on persons that have versions in French, English, and Spanish. WikiParq is available in six language versions and is potentially extendible to all the languages of Wikipedia. The WikiParq files are downloadable as tarball archives from this location: http://semantica.cs.lth.se/wikiparq/. Keywords: Wikipedia, Parquet, query language. 1. Introduction 1.1. Wikipedia Wikipedia is a collaborative, multilingual encyclopedia with more than 35 million articles across nearly 300 lan- guages. Anyone can edit it and its modification rate is of about 10 million edits per month1; many of its articles are categorized; and its cross-references to other articles (links) are extremely useful to help name disambiguation. Wikipedia is freely available in the form of dump archives2 and its wealth of information has made it a major resource in natural language processing applications that range from word counting to question answering (Ferrucci, 2012). 1.2. Wikipedia Annotation Wikipedia uses the so-called wiki markup to annotate the documents. Parsing this markup is then a compulsory step to any further processing. As for the Wikipedia ar- ticles, parts of this markup are language-dependent and can be created and changed by anyone. For exam- ple, the {{Citation needed}} template in the English Wikipedia is rendered by {{Citation nécessaire}} in French and {{citazione necessaria}} in Italian. Many articles in French use a date template in the form of {{Year|Month|Day}}, which is not used in other Wikipedias. Moreover, the wiki markup is difficult to master for the mil- lions of Wikipedia editors and, as a consequence, the arti- cles contain scores of malformed expressions. While it is relatively easy to create a quick-and-dirty parser, an accu- rate tool, functional across all the language versions is a ma- jor challenge. In this paper, we describe WikiParq, a set of Wikipedia archives with an easy tabular access. To create WikiParq, we reformatted Wikipedia dumps from their HTML render- ing and …

Content truncated to 500 words