Words and Echoes: Assessing and Mitigating the Non-Randomness Problem in Word Frequency Distribution Modeling

Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 904–911, Prague, Czech Republic, June 2007. c⃝2007 Association for Computational Linguistics Words and Echoes: Assessing and Mitigating the Non-Randomness Problem in Word Frequency Distribution Modeling Marco Baroni CIMeC (University of Trento) C.so Bettini 31 38068 Rovereto, Italy marco.baroni@unitn.it Stefan Evert IKW (University of Osnabr¨uck) Albrechtstr. 28 49069 Osnabr¨uck, Germany stefan.evert@uos.de Abstract Frequency distribution models tuned to words and other linguistic events can pre- dict the number of distinct types and their frequency distribution in samples of arbi- trary sizes. We conduct, for the first time, a rigorous evaluation of these models based on cross-validation and separation of train- ing and test data. Our experiments reveal that the prediction accuracy of the models is marred by serious overfitting problems, due to violations of the random sampling as- sumption in corpus data. We then propose a simple pre-processing method to allevi- ate such non-randomness problems. Further evaluation confirms the effectiveness of the method, which compares favourably to more complex correction techniques. 1 Introduction Large-Number-of-Rare-Events (LNRE) models (Baayen, 2001) are a class of specialized statistical models that allow us to estimate the characteristics of the distribution of type probabilities in type-rich linguistic populations (such as words) from limited samples (our corpora). They also allow us to extrapolate quantities such as vocabulary size (the number of distinct types) and the number of hapaxes (types occurring just once) beyond a given corpus or make predictions for completely unseen data from the same underlying population. LNRE models have applications in theoretical lin- guistics, e.g. for comparing the type richness of mor- phological or syntactic processes that are attested to different degrees in the data (Baayen, 1992). Con- sider for example a very common prefix such as re- and a rather rare prefix such as meta-. With LNRE models we can answer questions such as: If we could obtain as many tokens of meta- as we have of re-, would we also see as many distinct types? In other words, is the prefix meta- as productive as the prefix re-? Practical NLP applications, on the other hand, include estimating how many out-of- vocabulary words we will encounter given a lexicon of a certain size, or making informed guesses about type counts in very large data sets (e.g., how many typos are there on the Internet?) In this paper, after introducing LNRE models (Section 2), we present an evaluation of their per- formance based on separate training and test data as well as cross-validation (Section 3). As far as we know, this is the first time that such a rigorous evaluation has been conducted. The results show how evaluating on the training set, a common strat- egy in LNRE research, favours models that overfit the training data and perform poorly on unseen data. They also confirm the observation by Evert and Ba- roni (2006) that current LNRE models achieve only unsatisfactory prediction accuracy, and this is the is- sue we turn to in the second …

Content truncated to 500 words