Building a Web-Scale Dependency-Parsed Corpus from CommonCrawl

Building a Web-Scale Dependency-Parsed Corpus from Common Crawl Alexander Panchenko‡, Eugen Ruppert‡, Stefano Faralli†, Simone P. Ponzetto†, Chris Biemann‡ ‡ University of Hamburg, Department of Informatics, Language Technology Group, Germany † University of Mannheim, School of Business Informatics and Mathematics, Data and Web Science Group, Germany {panchenko,ruppert,biemann}@informatik.uni-hamburg.de {stefano,simone}@informatik.uni-mannheim.de Abstract We present DEPCC, the largest-to-date linguistically analyzed corpus in English including 365 million documents, composed of 252 billion tokens and 7.5 billion of named entity occurrences in 14.3 billion sentences from a web-scale crawl of the COMMON CRAWL project. The sentences are processed with a dependency parser and with a named entity tagger and contain provenance information, enabling various applications ranging from training syntax-based word embeddings to open information extraction and question answering. We built an index of all sentences and their linguistic meta-data enabling quick search across the corpus. We demonstrate the utility of this corpus on the verb similarity task by showing that a distributional model trained on our corpus yields better results than models trained on smaller corpora, like Wikipedia. This distributional model outperforms the state of art models of verb similarity trained on smaller corpora on the SimVerb3500 dataset. Keywords: text corpus, Web as a corpus, Common Crawl, dependency parsing, verb similarity, distributional semantics 1. Introduction Large corpora are essential for the modern data-driven approaches to natural language processing (NLP), espe- cially for unsupervised methods, such as word embed- dings (Mikolov et al., 2013) or open information extrac- tion (Banko et al., 2007) due to the “unreasonable effec- tiveness of big data” (Halevy et al., 2009). However, the size of commonly used text collections in the NLP com- munity, such as BNC1 or Wikipedia is in the range 0.1–3 billion tokens, which potentially limits coverage and per- formance of the developed models. To overcome this lim- itation, larger corpora can be composed of books, e.g. in (Goldberg and Orwant, 2013) a dataset of syntactic n- grams2 was built from the 345 billion token corpus of the Google Books project.3 However, access to books is often restricted, which limits use-cases of book-derived datasets. Another source of large amounts of texts is the Web. Multiple researchers investigated the use of the Web for construction of text corpora, producing resources, such as PUKWAC (Baroni et al., 2009) (2 billion of tokens) and ENCOW16 (Sch¨afer, 2015) (17 billion of tokens), yet the size of these corpora is still at least one order of magni- tude smaller than the web-scale crawls, e.g. CLUEWEB4 and COMMON CRAWL5. On the other hand, directly using the web crawl dumps is problematic for researchers as: (1) the documents are not preprocessed, containing irrelevant information, e.g. HTML markup; (2) big data infrastruc- ture and skills are required; (3) (near)duplicates of pages disbalance the corpus; (4) documents are not linguistically analyzed, thus only shallow models can be used. The men- tioned factors substantially limit the use of web-scale cor- 1http://www.natcorp.ox.ac.uk 2http://commondatastorage.googleapis.com/ books/syntactic-ngrams/index.html 3https://books.google.com 4http://lemurproject.org/clueweb12 5http://www.commoncrawl.org pora in natural language processing research and applica- tions. The objective of this …

Content truncated to 500 words