A Large Scale Evaluation of Distributional Semantic Models: Parameters, Interactions and Model Selection

A Large Scale Evaluation of Distributional Semantic Models: Parameters, Interactions and Model Selection Gabriella Lapesa2,1 1Universit¨at Osnabr¨uck Institut f¨ur Kognitionswissenschaft Albrechtstr. 28, Osnabr¨uck, Germany gabriella.lapesa@fau.de Stefan Evert2 2FAU Erlangen-N¨urnberg Professur f¨ur Korpuslinguistik Bismarckstr. 6, Erlangen, Germany stefan.evert@fau.de Abstract This paper presents the results of a large-scale evaluation study of window-based Distribu- tional Semantic Models on a wide variety of tasks. Our study combines a broad coverage of model parameters with a model selection methodology that is robust to overfitting and able to capture parameter interactions. We show that our strategy allows us to identify pa- rameter configurations that achieve good per- formance across different datasets and tasks1. 1 Introduction Distributional Semantic Models (DSMs) are em- ployed to produce semantic representations of words from co-occurrence patterns in texts or documents (Sahlgren, 2006; Turney and Pantel, 2010). Build- ing on the Distributional Hypothesis (Harris, 1954), DSMs quantify the amount of meaning shared by words as the degree of overlap of the sets of contexts in which they occur. A widely used approach operationalizes the set of contexts as co-occurrences with other words within a certain window (e.g., 5 words). A window-based DSM can be represented as a co-occurrence matrix in which rows correspond to target words, columns correspond to context words, and cells store the co- occurrence frequencies of target words and context words. The co-occurrence information is usually weighted by some scoring function and the rows of the matrix are normalized. Since the co-occurrence 1The analysis presented in this paper is complemented by supplementary materials, which are available for download at http://www.linguistik.fau.de/dsmeval/. This page will also be kept up to date with the results of follow-up experiments. matrix tends to be very large and sparsely popu- lated, dimensionality reduction techniques are often used to obtain a more compact representation. Lan- dauer and Dumais (1997) claim that dimensionality reduction also improves the semantic representation encoded in the co-occurrence matrix. Finally, dis- tances between the row vectors of the matrix are computed and – according to the Distributional Hy- pothesis – interpreted as a correlate of the semantic similarities between the corresponding target words. The construction and use of a DSM involves many design choices, such as: selection of a source cor- pus, size of the co-occurrence window; choice of a suitable scoring function, possibly combined with an additional transformation; whether to apply dimen- sionality reduction, and the number of reduced di- mensions; metric for measuring distances between vectors. Different design choices – technically, the DSM parameters – can result in quite different sim- ilarities for the same words (Sahlgren, 2006). DSMs have already proven successful in model- ing lexical meaning: they have been applied in Natu- ral Language Processing (Sch¨utze, 1998; Lin, 1998), Information Retrieval (Salton et al., 1975), and Cog- nitive Modeling (Landauer and Dumais, 1997; Lund and Burgess, 1996; Pad´o and Lapata, 2007; Ba- roni and Lenci, 2010). Recently, the field of Dis- tributional Semantics has moved towards new chal- lenges, such as predicting brain activation (Mitchell et al., 2008; …

Content truncated to 500 words