Using Collections of Human Language Intuitions to Measure Corpus Representativeness

Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 2117–2128, Dublin, Ireland, August 23-29 2014. Using Collections of Human Language Intuitions to Measure Corpus Representativeness Reinhard Rapp Aix-Marseille Université Laboratoire d'Informatique Fondamentale 163 Avenue de Luminy, 13288 Marseille, France reinhardrapp@gmx.de Abstract In corpus linguistics there have been numerous attempts to compile balanced corpora, result- ing in text collections such as the Brown Corpus or the British National Corpus. These cor- pora are meant to reflect the average language use a native speaker typically encounters. But is it possible to measure in how far these efforts were successful? Assuming that humans’ lan- guage intuitions are based on our brain’s capability to statistically analyze perceived language and to memorize these statistics, we suggest a method for measuring corpus representative- ness which compares corpus statistics to three types of human language intuitions as collected from test persons: Word familiarity, word association, and word relatedness. We compute a representativeness score for a corpus by extracting word frequency, word co-occurrence, and contextual statistics from it and by comparing these statistics to the human data. The higher the similarity, the more representative the corpus should be for the language environments of the test persons. Our findings confirm the expectation that corpus size and corpus balancing matter. 1 Introduction Balanced corpora, i.e. corpora consisting of a carefully sampled mix of texts, have often been consid- ered important for providing a standard of average language use. Well known examples of such cor- pora include the Brown Corpus (Francis & Kuςera, 1989) and the British National Corpus (Burnard & Aston, 1998). But to obtain a balance many decisions concerning the corpus design have to be made. Biber (1993) mentions, among other things, that it has to be decided for what target population a cor- pus is meant to be representative, that estimates concerning the quantities of various text types are re- quired, and that decisions with regard to the number of individual text samples and their sizes have to be made. However, there is no easy and well established way to verify the success of these measures. Current suggestions include, for example, to consider a corpus as representative if it is not dominated by sub- language (Temnikova et al., 2014), or to more or less give up on the concept of representativeness and to concentrate on considering the suitability of a corpus for particular tasks. Saldanha (2009) comes to the conclusion that “The problem with making representativeness the defining characteristic of a cor- pus is that it is very difficult to evaluate.ˮ Our goal here is to make an attempt to measure corpus representativeness in a standardized way, thereby avoiding to observe test persons’ average language input as this would not be very practical. Our starting point is that a representative corpus should reflect as well as possible average language use as encountered by native speakers. We also assume that human language acquisition is essentially corpus-based (Rapp, 2011). This implies the following: The human …

Content truncated to 500 words