Global Evaluation of Random Indexing through Swedish Word Clustering Compared to the People’s Dictionary of Synonyms

International Conference RANLP 2009 - Borovets, Bulgaria, pages 376–380 Global Evaluation of Random Indexing through Swedish Word Clustering Compared to the People’s Dictionary of Synonyms Magnus Rosell KTH CSC Stockholm, Sweden rosell@csc.kth.se Martin Hassel DSV, KTH - Stockholm University Kista, Sweden xmartin@dsv.su.se Viggo Kann KTH CSC Stockholm, Sweden viggo@nada.kth.se Abstract Evaluation of word space models is usually local in the sense that it only considers words that are deemed very similar by the model. We propose a global evaluation scheme based on clustering of the words. A clustering of high quality in an external evaluation against a semantic resource, such as a dictionary of synonyms, indicates a word space model of high quality. We use Random Indexing to create several dif- ferent models and compare them by cluster- ing evaluation against the People’s Dictionary of Synonyms, a list of Swedish synonyms that are graded by the public. Most notably we get better results for models based on syntagmatic information (words that appear together) than for models based on paradigmatic information (words that appear in similar contexts). This is quite contrary to previous results that have been presented for local evaluation. Clusterings to ten clusters result in a recall of 83% for a syntagmatic model, compared to 34% for a comparable paradigmatic model, and 10% for a random partition. Keywords Random Indexing, Word Space Model, Word Clustering, Eval- uation, Dictionary of Synonyms 1 Introduction Word space models (see among others [1, 16, 11, 6, 15]) map words to vectors in a multidimensional space by extracting statistics about the context they appear in from a large sample of text. Words that thus become represented by similar vectors (as measured by a simi- larity measure such as the cosine measure) are consid- ered related. What this (meaning) relation could be referred to in ordinary (human) semantics is not ob- vious. It may capture something like synonymy, but may as well regard for instance antonyms, and a hy- ponym and its hyperonym as highly related. Relations between words based on their contexts can be divided into two categories [15]: Two words have a relation that is syntagmatic if they appear together. paradigmatic if they appear in similar contexts. Word space models can be constructed in attempts to capture either of these two relations. In this work we use Random Indexing (see Section 2) to construct several different word space models. Word space models have been evaluated using sev- eral different schemes [15]. They are all local in that they only consider a small part of the words in the model. We introduce a new global evaluation scheme that takes all words in the model into consideration, using word clustering and a list of synonyms. The paper is organized as follows. Sections 2 and 3 describe Random Indexing and word clustering. We discuss evaluation of word space models in general and present our proposed global evaluation scheme in Sec- tion 4. In Section 5 we describe and discuss our ex- periments: the text set we …

Content truncated to 500 words