Words and Word Usage: Newspaper Text versus the Web

Proceedings of the Australasian Language Technology Workshop 2005, pages 167–175, Sydney, Australia, December 2005. Words and Word Usage: Newspaper Text versus the Web Vinci Liu and James R. Curran School of Information Technologies University of Sydney NSW 2006, Australia {vinci,james}@it.usyd.edu.au Abstract This paper explores the differences in words and word usage in two corpora – one derived from newspaper text and the other from the web. A corpus of web pages is compiled from a con- trolled traversal of the web, producing a topic- diverse collection of 2 billion words of web text1. We compare this Web Corpus with the Giga- word Corpus, a 2 billion word collection of news articles. The Web Corpus is applied to the task of automatic thesaurus extraction, obtain- ing similar overall results to using the Gigaword. The quality of synonyms extracted for each tar- get word is dependent on the word’s usage in the corpus. With many more words available on the web, a much larger Web Corpus can be created to obtain better results in different nlp tasks. 1 Introduction In corpus-based Natural Language Processing (nlp), the corpus is the primary representation of a language from which algorithms extract in- formation and build linguistic models. Words and word usage in a corpus of newspaper text will differ from that of a corpus of web pages, due to the different genres of text and as an artifact of the corpus collection process. Words are used differently across different medium and corpora. While newspaper text is usually written by experienced writers and carefully checked by editors for accuracy, few restrictions exist for web text. It has a wider range of writing styles and less adherence to for- mal grammar. Anyone with access to the web can create web pages and there is no restriction as to what topics are written about on the web. Thus, a web corpus will contain a wider range of topics than a newspaper corpus. 1In this paper, we report on the number of tokens after the corpus has been tokenised, counting both words and punctuation. Some differences between corpora can be at- tributed to the collection process. A corpus of newspaper text is usually collected from a few publishers across a set period of time. This cor- pus will reflect the topics in the news over that period as well as the types of news targeted by the publishers (e.g. political, financial). A cor- pus of web pages can contain pages from any time before its creation. It will have a different distribution of dates than a newspaper corpus. In this paper, we explore the difference be- tween a corpus of newspaper text and a corpus of web text. We conduct three experiments to highlight some of the differences. First, the to- ken types that exist within each corpus are ex- amined and the vocabulary unique to each cor- pus is accounted for. We then analysed words frequent in one corpus that are infrequent in the other to reveal topic …

Content truncated to 500 words