LanguageCrawl: A Generic Tool for Building Language Models Upon Common-Crawl

LanguageCrawl: A Generic Tool for Building Language Models Upon Common-Crawl Szymon Roziewski, Wojciech Stokowiec National Information Processing Institute Natural Language Processing Laboratory, Warsaw, Poland sroziewski@opi.org.pl, wstokowiec@opi.org.pl Abstract The web data contains immense amount of data, hundreds of billion words are waiting to be extracted and used for language research. In this work we introduce our tool LanguageCrawl which allows Natural Language Processing (NLP) researchers to easily construct web-scale corpus the from Common Crawl Archive: a petabyte scale open repository of web crawl information. Three use-cases are presented: filtering Polish websites, building N-gram corpora and training continuous skip-gram language model with hierarchical softmax. Each of them has been implemented within the LanguageCrawl toolkit, with the possibility to adjust specified language and N-gram ranks. Special effort has been put on high computing efficiency, by applying highly concurrent multitasking. We make our tool publicly available to enrich NLP resources. We strongly believe that our work will help to facilitate NLP research, especially in under-resourced languages, where the lack of appropriately sized corpora is a serious hindrance to applying data-intensive methods, such as deep neural networks. Keywords: Polish Web Corpus, Common Crawl, Word2Vec 1. Introduction The Internet is the largest and most diverse collection of textual information in human history, it covers almost all known subjects and languages. It constitutes an appealing resource for extraction of large-scale corpora for language modelling. However, until recently, it was highly unlikely that language researchers in the academia would have had access to the necessary infrastructure needed to process the Internet in an effort to build a language corpus. With the re- cent improvements of computing power, storage availabil- ity and powerful, highly efficient scalable processing and computing frameworks, it has become feasible to build a large scale corpus using commodity hardware1 and publicly available web-archives. Our tool LanguageCrawl enables NLP researchers to eas- ily construct large-scale corpus of a given language filtered directly from the Common Crawl Archive. We believe that the linguistic community could benefit from our work: precomputed N-grams or distributed word representations could be used for example to boost the accuracy of machine translation systems (Buck et al., 2014). In this work our primary objective is to illustrate the use cases for LanguageCrawl: filtering Polish Websites from the Common Crawl Archive and subsequently building N- gram corpora and training continuous Skip-gram language model. As far as we know, nobody has formed out such a collection made from the Polish Internet. We are interested in Polish language since we would like to use our results for further NLP research for Polish, and to enrich general knowledge about it, i.e. models and data. Both, source code and the language models will be made publicly available by the time this paper is published. 1By this term we understand a piece of fairly standard hard- ware that can be purchased at retail, such as low-cost desktop computers. 1.1. Data Set Common Crawl Archive2, on which our tool has been based, is an open repository of web crawl information …

Content truncated to 500 words