A Very Very Large Corpus Doesn’t Always Yield Reliable Estimates

A very very large corpus doesn’t always yield reliable estimates James R. Curran and Miles Osborne Institute for Communicating and Collaborative Systems University of Edinburgh 2 Buccleuch Place, Edinburgh EH8 9LW United Kingdom � jamesc,osborne � @cogsci.ed.ac.uk Abstract Banko and Brill (2001) suggested that the develop- ment of very large training corpora may be more ef- fective for progress in empirical Natural Language Processing than improving methods that use exist- ing smaller training corpora. This work tests their claim by exploring whether a very large corpus can eliminate the sparseness problems associated with estimating unigram prob- abilities. We do this by empirically investigating the convergence behaviour of unigram probability estimates on a one billion word corpus. When us- ing one billion words, as expected, we do find that many of our estimates do converge to their eventual value. However, we also find that for some words, no such convergence occurs. This leads us to con- clude that simply relying upon large corpora is not in itself sufficient: we must pay attention to the sta- tistical modelling as well. 1 Introduction The quantity and reliability of linguistic informa- tion is primarily determined by the size of the train- ing corpus: with limited data available, extracting statistics for any given language phenomenon and its surrounding context is unreliable. Overcoming the sparse distribution of linguistic events is a key design problem in any statistical NLP system. For some tasks, corpus size is no longer a limit- ing factor, since it has become feasible to acquire homogeneous document collections two or three or- ders of magnitude larger than existing resources. Banko and Brill (2001) report on confusion set disambiguation experiments where they apply rela- tively simple learning methods to a one billion word training corpus. Their experiments show a logarith- mic trend in performance as corpus size increases without performance reaching an upper bound. This leads them to believe that the development of large scale training material will yield superior results than further experimentation with machine learning methods on existing smaller scale training corpora. Recent work has replicated the Banko and Brill (2001) results on the much more complex task of automatic thesaurus extraction, showing that con- textual statistics, collected over a very large corpus, significantly improve system performance (Curran and Moens, 2002). Other research has shown that query statistics from a web search engine can be used as a substitute for counts collected from large corpora (Volk, 2001; Keller et al., 2002). To further investigate the benefits of using very large corpora we empirically analyse the conver- gence behaviour of unigram probability estimates for a range of words with different relative frequen- cies. By dramatically increasing the size of the training corpus, we expect our confidence in the probability estimates for each word to increase. As theory predicts, unigram probability estimates for many words do converge as corpus size grows. However, contrary to intuition, we found that for many commonplace words, for example tight- ness, there was no sign of convergence as corpus …

Content truncated to 500 words