Don’t count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors

Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, pages 238–247, Baltimore, Maryland, USA, June 23-25 2014. c⃝2014 Association for Computational Linguistics Don’t count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors Marco Baroni and Georgiana Dinu and Germ´an Kruszewski Center for Mind/Brain Sciences (University of Trento, Italy) (marco.baroni|georgiana.dinu|german.kruszewski)@unitn.it Abstract Context-predicting models (more com- monly known as embeddings or neural language models) are the new kids on the distributional semantics block. Despite the buzz surrounding these models, the litera- ture is still lacking a systematic compari- son of the predictive models with classic, count-vector-based distributional semantic approaches. In this paper, we perform such an extensive evaluation, on a wide range of lexical semantics tasks and across many parameter settings. The results, to our own surprise, show that the buzz is fully justified, as the context-predicting models obtain a thorough and resounding victory against their count-based counter- parts. 1 Introduction A long tradition in computational linguistics has shown that contextual information provides a good approximation to word meaning, since semanti- cally similar words tend to have similar contex- tual distributions (Miller and Charles, 1991). In concrete, distributional semantic models (DSMs) use vectors that keep track of the contexts (e.g., co-occurring words) in which target terms appear in a large corpus as proxies for meaning represen- tations, and apply geometric techniques to these vectors to measure the similarity in meaning of the corresponding words (Clark, 2013; Erk, 2012; Turney and Pantel, 2010). It has been clear for decades now that raw co- occurrence counts don’t work that well, and DSMs achieve much higher performance when various transformations are applied to the raw vectors, for example by reweighting the counts for con- text informativeness and smoothing them with di- mensionality reduction techniques. This vector optimization process is generally unsupervised, and based on independent considerations (for ex- ample, context reweighting is often justified by information-theoretic considerations, dimension- ality reduction optimizes the amount of preserved variance, etc.). Occasionally, some kind of indi- rect supervision is used: Several parameter set- tings are tried, and the best setting is chosen based on performance on a semantic task that has been selected for tuning. The last few years have seen the development of a new generation of DSMs that frame the vec- tor estimation problem directly as a supervised task, where the weights in a word vector are set to maximize the probability of the contexts in which the word is observed in the corpus (Bengio et al., 2003; Collobert and Weston, 2008; Collobert et al., 2011; Huang et al., 2012; Mikolov et al., 2013a; Turian et al., 2010). The traditional con- struction of context vectors is turned on its head: Instead of first collecting context vectors and then reweighting these vectors based on various crite- ria, the vector weights are directly set to optimally predict the contexts in which the corresponding words tend to appear. Since similar words occur in similar contexts, the system naturally learns to assign similar vectors to similar …

Content truncated to 500 words