A Figure of Merit for the Evaluation of Web-Corpus Randomness

A Figure of Merit for the Evaluation of Web-Corpus Randomness Massimiliano Ciaramita Institute of Cognitive Science and Technology National Research Council Roma, Italy m.ciaramita@istc.cnr.it Marco Baroni SSLMIT Universit`a di Bologna Forl`ı, Italy baroni@sslmit.unibo.it Abstract In this paper, we present an automated, quantitative, knowledge-poor method to evaluate the randomness of a collection of documents (corpus), with respect to a number of biased partitions. The method is based on the comparison of the word frequency distribution of the target corpus to word frequency distributions from cor- pora built in deliberately biased ways. We apply the method to the task of building a corpus via queries to Google. Our results indicate that this approach can be used, reliably, to discriminate biased and unbi- ased document collections and to choose the most appropriate query terms. 1 Introduction The Web is a very rich source of linguistic data, and in the last few years it has been used in- tensively by linguists and language technologists for many tasks (Kilgarriff and Grefenstette, 2003). Among other uses, the Web allows fast and in- expensive construction of “general purpose” cor- pora, i.e., corpora that are not meant to repre- sent a specific sub-language, but a language as a whole. There are several recent studies on the extent to which Web-derived corpora are com- parable, in terms of variety of topics and styles, to traditional “balanced” corpora (Fletcher, 2004; Sharoff, 2006). Our contribution, in this paper, is to present an automated, quantitative method to evaluate the “variety” or “randomness” (with re- spect to a number of non-random partitions) of a Web corpus. The more random/less-biased to- wards specific partitions a corpus is, the more it should be suitable as a general purpose corpus. We are not proposing a method to evaluate whether a sample of Web pages is a random sam- ple of the Web, although this is a related issue (Bharat and Broder, 1998; Henzinger et al., 2000). Instead, we propose a method, based on simple distributional properties, to evaluate if a sample of Web pages in a certain language is reasonably varied in terms of the topics (and, perhaps, tex- tual types) it contains. This is independent from whether they are actually proportionally represent- ing what is out there on the Web or not. For exam- ple, although computer-related technical language is probably much more common on the Web than, say, the language of literary criticism, one might prefer a biased retrieval method that fetches docu- ments representing these and other sub-languages in comparable amounts, to an unbiased method that leads to a corpus composed mostly of com- puter jargon. This is a new area of investigation – with traditional corpora, one knows a priori their composition. As the Web plays an increasingly central role as data source in NLP, we believe that methods to efficiently characterize the nature of automatically retrieved data are becoming of cen- tral importance to the discipline. In the empirical evaluation of the method, we focus on general purpose corpora built issuing …

Content truncated to 500 words