The Effects of Data Size and Frequency Range on Distributional Semantic Models

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 975–980, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics The Effects of Data Size and Frequency Range on Distributional Semantic Models Magnus Sahlgren Gavagai and SICS Slussplan 9, Box 1263 111 30 Stockholm, 164 29 Kista Sweden mange@[gavagai|sics].se Alessandro Lenci University of Pisa via Santa Maria 36 56126 Pisa Italy alessandro.lenci@unipi.it Abstract This paper investigates the effects of data size and frequency range on distributional seman- tic models. We compare the performance of a number of representative models for several test settings over data of varying sizes, and over test items of various frequency. Our re- sults show that neural network-based models underperform when the data is small, and that the most reliable model over data of varying sizes and frequency ranges is the inverted fac- torized model. 1 Introduction Distributional Semantic Models (DSMs) have be- come a staple in natural language processing. The various parameters of DSMs — e.g. size of con- text windows, weighting schemes, dimensionality reduction techniques, and similarity measures — have been thoroughly studied (Weeds et al., 2004; Sahlgren, 2006; Riordan and Jones, 2011; Bulli- naria and Levy, 2012; Levy et al., 2015), and are now well understood. The impact of various pro- cessing models — matrix-based models, neural net- works, and hashing methods — have also enjoyed considerable attention lately, with at times conflict- ing conclusions (Baroni et al., 2014; Levy et al., 2015; Schnabel et al., 2015; ¨Osterlund et al., 2015; Sahlgren et al., 2016). The consensus interpretation of such experiments seems to be that the choice of processing model is less important than the parame- terization of the models, since the various processing models all result in more or less equivalent DSMs (provided that the parameterization is comparable). One of the least researched aspects of DSMs is the effect on the various models of data size and frequency range of the target items. The only pre- vious work in this direction that we are aware of is Asr et al. (2016), who report that on small data (the CHILDES corpus), simple matrix-based mod- els outperform neural network-based ones. Unfor- tunately, Asr et al. do not include any experiments using the same models applied to bigger data, mak- ing it difficult to compare their results with previous studies, since implementational details and parame- terization will be different. There is thus still a need for a consistent and fair comparison of the performance of various DSMs when applied to data of varying sizes. In this pa- per, we seek an answer to the question: which DSM should we opt for if we only have access to lim- ited amounts of data? We are also interested in the related question: which DSM should we opt for if our target items are infrequent? The latter ques- tion is particularly crucial, since one of the major as- sets of DSMs is their applicability to create seman- tic representations for ever-expanding vocabularies from text …

Content truncated to 500 words