C4Corpus: Multilingual Web-size Corpus with Free License

C4Corpus: Multilingual Web-size Corpus with Free License Ivan Habernal†‡, Omnia Zayed‡, Iryna Gurevych†‡ † Ubiquitous Knowledge Processing Lab ‡ Research Training Group AIPHES Department of Computer Science, Technische Universit¨at Darmstadt, Hochschulstr. 10, 64289 Darmstadt, Germany www.ukp.tu-darmstadt.de, www.aiphes.tu-darmstadt.de Abstract Large Web corpora containing full documents with permissive licenses are crucial for many NLP tasks. In this article we present the construction of 12 million-pages Web corpus (over 10 billion tokens) licensed under CreativeCommons license family in 50+ languages that has been extracted from CommonCrawl, the largest publicly available general Web crawl to date with about 2 billion crawled URLs. Our highly-scalable Hadoop-based framework is able to process the full CommonCrawl corpus on 2000+ CPU cluster on the Amazon Elastic Map/Reduce infrastructure. The processing pipeline includes license identification, state-of-the-art boilerplate removal, exact duplicate and near-duplicate document removal, and language detection. The construction of the corpus is highly configurable and fully reproducible, and we provide both the framework (DKPro C4CorpusTools) and the resulting data (C4Corpus) to the research community. Keywords: CommonCrawl, Creative Commons, Web corpus, Amazon Web Services 1. Introduction Availability of large-scale corpora is crucial for state-of- the-art Natural Language Processing (NLP). The impor- tance of both annotated and raw large-scale corpora is rapidly increasing due to recent success of neural networks and similar semi- or unsupervised methods in a wide variety of language processing tasks. In recent years, tremendous progress has been made with sentence-level tasks (such as dependency parsing) and genre-specific benchmarks (such as work on the Penn Discourse Treebank). There is also an increasing demand for solutions scaling to heteroge- neous document collections on the web. Current trends lean toward multilingual solutions, e.g., universal POS tags (Petrov et al., 2012), which requires heterogeneous cor- pora in multiple languages. Furthermore, recent document- level research tasks, such as multi-document summariza- tion (Bing et al., 2015) or argumentation analysis (Habernal and Gurevych, 2015), heavily depend on document-level training and evaluation corpora. One of the big obstacles for the current research is the lack of large-scale freely-licensed heterogeneous corpora in multiple languages, which can be re-distributed in the form of entire documents. Existing corpora are limited along several dimensions. First, they often exhibit mono- lingual nature, e.g., ClueWeb1, Annotated English Giga- word2 (Napoles et al., 2012), and several *WaC corpora (Ljubeˇsi´c and Klubiˇcka, 2014; Faaß and Eckart, 2013). Second, they are usually available as either n-grams (Brants and Franz, 2006) or shuffled sentences, e.g., COW (Sch¨afer and Bildhauer, 2012) or Leipzig Corpora (Goldhahn et al., 2012). Third, some corpora cover only a limited range of genres, e.g., discussions (Hl´adek et al., 2014), newswire (Spoustov´a and Spousta, 2012), or Wiki-texts (Lyding et al., 2014). Finally, due to the restrictive license of the content, many corpora cannot be re-distributed because of the risk of copyright infringement (Biemann et al., 2013; 1http://www.lemurproject.org/clueweb12/ 2https://catalog.ldc.upenn.edu/LDC2012T21 Sch¨afer, 2015). To the best of our knowledge, no current approaches tar- get at filling this gap. In order to scale up to the Inter- net size, such an approach would require state-of-the-art …

Content truncated to 500 words