CommonCOW: Massively Huge Web Corpora from CommonCrawl Data and a Method to Distribute them Freely under Restrictive EU Copyright Laws

CommonCOW: Massively Huge Web Corpora from CommonCrawl Data and a Method to Distribute them Freely under Restrictive EU Copyright Laws Roland Schäfer German Grammar Freie Universität Berlin Habelschwerdter Allee 45, 14195 Berlin roland.schaefer@fu-berlin.de Abstract In this paper, I describe a method of creating massively huge web corpora from the CommonCrawl data sets and redistributing the result- ing annotations in a stand-off format. Current EU (and especially German) copyright legislation categorically forbids the redistribution of downloaded material without express prior permission by the authors. Therefore, stand-off annotations or other derivates are the only format in which European researchers (like myself) are allowed to re-distribute the respective corpora. In order to make the full corpora available to the public despite such restrictions, the stand-off format presented here allows anybody to locally reconstruct the full corpora with the least possible computational effort. In Section 1., I briefly introduce the technology behind the COW project (Corpora from the Web), which is used to create the CommonCrawl-derived corpora. In Section 2., I provide some details about the resulting CommonCOW (COCO) web corpora. Finally, in Section 3., I introduce a method to circumvent restrictive EU copyright laws by distributing only the corpus annotations (under a CC-BY license) together with a tool that allows users to locally reconstruct the corpus from the annotations and the original CommonCrawl files. Keywords: web corpora, stand-off annotation, legal issues 1. Web corpora and COW Over the past decade, web corpora have been created by various research groups (Biemann et al., 2007; Baroni et al., 2009; Pomikalek et al., 2009; Schäfer and Bildhauer, 2012; Schäfer and Bildhauer, 2013), and they have been actively used in Computational Linguistics and Corpus Linguistics. The COCO project introduced in this paper uses the exist- ing technology developed by the COW (Corpora from the Web) project at Freie Universität Berlin (Schäfer and Bild- hauer, 2012; Schäfer et al., 2013) in the COW14 version (Schäfer, 2015b).1 The backbone of this technology is the texrex web page cleaning tool.2 Using a bundle of computational methods described in previous publications, texrex (in its 2015 ver- sion texrex-behindthecow): 1. filters out perfect and near-duplicate documents (Schäfer and Bildhauer, 2012), 2. strips HTML markup and scripts, 3. converts encodings (to UTF-8) and performs NFC Unicode normalization, 4. detects and annotates the document language, 5. classifies paragraphs as boilerplate or good text (Schäfer, 2015a), 6. classifies documents as containing more or less coher- ent text (Schäfer et al., 2013), 7. extracts meta information from the crawl headers and the HMTL source, 8. performs IP-based server geolocation. Linguistic annotation (for Dutch, English, French, German, Spanish, Swedish) is done using available tools. For ex- ample, English COW corpora are tokenized using Ucto 1http://corporafromtheweb.org/ 2http://texrex.sourceforge.net/ (van Gompel et al., 2012), part-of-speech tagged and chun- ked using TreeTagger (Schmid, 1994), and dependency parsed with the Malt Parser (Nivre et al., 2007). German is tokenized, tagged, and chunked using the same tools, named entity recognition is performed using the Stanford NER tool and available German models (Faruqui and …

Content truncated to 500 words