Domain-Specific Corpus Expansion with Focused Webcrawling

Domain-Specific Corpus Expansion with Focused Webcrawling Steffen Remus and Chris Biemann FG Language Technology Computer Science Department Technische Universität Darmstadt, Darmstadt, Germany {remus,biem}@cs.tu-darmstadt.de Abstract This work presents a straightforward method for extending or creating in-domain web corpora by focused webcrawling. The focused webcrawler uses statistical N-gram language models to estimate the relatedness of documents and weblinks and needs as input only N-grams or plain texts of a predefined domain and seed URLs as starting points. Two experiments demonstrate that our focused crawler is able to stay focused in domain and language. The first experiment shows that the crawler stays in a focused domain, the second experiment demonstrates that language models trained on focused crawls obtain better perplexity scores on in-domain corpora. We distribute the focused crawler as open source software. Keywords: web crawling, focused, topical, in-domain, web-corpora, language model, perplexity 1. Introduction With increasing power of computational resources and al- gorithms to efficiently process more and more data in less time, the demand for larger text collections grows and the web as a huge and dynamic resource is nowadays the main source of any kind of data. The WaCKy1 corpora (Baroni et al., 2009) or the COW2 project (Schäfer and Bildhauer, 2012) are just some examples for language- processing-oriented web corpora. ClueWeb (Callan et al., 2009) or Amazons common crawl (Common Crawl Foun- dation, 2011) provide unprocessed html data from web crawls, which are further refined e.g. in (Pomikálek et al., 2012; Buck et al., 2014). However, the data is largely collected without notions of topical interest. If an interest in a particular topic exists, corpora have to undergo extensive document filtering with simple and/or complex text classification methods. This leads to a lot of downloaded data being discarded with lots of computational resources being unnecessarily wasted. One approach to work around these issues is to use the BootCat method (Baroni and Bernardini, 2004), which col- lects, based on keyword lists, web documents by send- ing combinations of keywords to a search engine provider. Here, one particular disadvantage is the use of a search en- gine provider as a black box, which makes it dependent on a) the general availability of the service, and b) ranking of the results based on the provider’s (possibly subjective) choice (Kilgarriff, 2007). This paper introduces a simple tool for focused crawling, which makes efficient use of computational resources as it downloads mainly websites of interest, i.e. those belonging to a certain topic. The domain of interest is defined by a statistical N-gram language model or sample texts that can be used to create a language model from. 1Web-as-Corpus Kool Yinitiative http://wacky. sslmit.unibo.it/ 2Corpora from the web, http://corporafromtheweb. org/ 2. Focused Crawling The term ‘focused crawling’ (Chakrabarti et al., 1999) also known as ‘topical crawling’ (Menczer et al., 2004) refers to the process of crawling the web in a guided way with a focus on a specific topic. The task is to decide which link to follow and which not, or in which order to …

Content truncated to 500 words