Web Text Corpus for Natural Language Processing

Web Text Corpus for Natural Language Processing Vinci Liu and James R. Curran School of Information Technologies University of Sydney NSW 2006, Australia {vinci,james}@it.usyd.edu.au Abstract Web text has been successfully used as training data for many NLP applications. While most previous work accesses web text through search engine hit counts, we created a Web Corpus by downloading web pages to create a topic-diverse collec- tion of 10 billion words of English. We show that for context-sensitive spelling correction the Web Corpus results are bet- ter than using a search engine. For the- saurus extraction, it achieved similar over- all results to a corpus of newspaper text. With many more words available on the web, better results can be obtained by col- lecting much larger web corpora. 1 Introduction Traditional written corpora for linguistics research are created primarily from printed text, such as newspaper articles and books. With the growth of the World Wide Web as an information resource, it is increasingly being used as training data in Nat- ural Language Processing (NLP) tasks. There are many advantages to creating a corpus from web data rather than printed text. All web data is already in electronic form and therefore readable by computers, whereas not all printed data is available electronically. The vast amount of text available on the web is a major advantage, with Keller and Lapata (2003) estimating that over 98 billion words were indexed by Google in 2003. The performance of NLP systems tends to im- prove with increasing amount of training data. Banko and Brill (2001) showed that for context- sensitive spelling correction, increasing the train- ing data size increases the accuracy, for up to 1 billion words in their experiments. To date, most NLP tasks that have utilised web data have accessed it through search engines, us- ing only the hit counts or examining a limited number of results pages. The tasks are reduced to determining n-gram probabilities which are then estimated by hit counts from search engine queries. This method only gathers information from the hit counts but does not require the com- putationally expensive downloading of actual text for analysis. Unfortunately search engines were not designed for NLP research and the reported hit counts are subject to uncontrolled variations and approximations (Nakov and Hearst, 2005). Volk (2002) proposed a linguistic search engine to ex- tract word relationships more accurately. We created a 10 billion word topic-diverse Web Corpus by spidering websites from a set of seed URLs. The seed set is selected from the Open Directory to ensure that a diverse range of top- ics is included in the corpus. A process of text cleaning transforms the HTML text into a form useable by most NLP systems – tokenised words, one sentence per line. Text filtering removes un- wanted text from the corpus, such as non-English sentences and most lines of text that are not gram- matical sentences. We compare the vocabulary of the Web Corpus with newswire. Our Web Corpus is evaluated on two …

Content truncated to 500 words