Random Indexing using Statistical Weight Functions

Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing (EMNLP 2006), pages 457–464, Sydney, July 2006. c⃝2006 Association for Computational Linguistics Random Indexing using Statistical Weight Functions James Gorman and James R. Curran School of Information Technologies University of Sydney NSW 2006, Australia {jgorman2,james}@it.usyd.edu.au Abstract Random Indexing is a vector space tech- nique that provides an efficient and scal- able approximation to distributional simi- larity problems. We present experiments showing Random Indexing to be poor at handling large volumes of data and evalu- ate the use of weighting functions for im- proving the performance of Random In- dexing. We find that Random Index is ro- bust for small data sets, but performance degrades because of the influence high fre- quency attributes in large data sets. The use of appropriate weight functions im- proves this significantly. 1 Introduction Synonymy relations between words have been used to inform many Natural Language Processing (NLP) tasks. While these relations can be extracted from manually created resources such as thesauri (e.g. Roget’s Thesaurus) and lexical databases (e.g. WordNet, Fellbaum, 1998), it is often ben- eficial to extract these relationships from a corpus representative of the task. Manually created resources are expensive and time-consuming to create, and tend to suffer from problems of bias, inconsistency, and limited cover- age. These problems may result in an inappropri- ate vocabulary, where some terms are not present or an unbalanced set of synonyms. In a medical context it is more likely that administration will re- fer to the giving of medicine than to paper work, whereas in a business context the converse is more likely. The most common method for automatically creating these resources uses distributional simi- larity and is based on the distributional hypoth- esis that similar words appear in similar con- texts. Terms are described by collating informa- tion about their occurrence in a corpus into vec- tors. These context vectors are then compared for similarity. Existing approaches differ primarily in their definition of context, e.g. the surrounding words or the entire document, and their choice of distance metric for calculating similarity between the context vectors representing each term. In this paper, we analyse the use of Random In- dexing (Kanerva et al., 2000) for semantic similar- ity measurement. Random Indexing is an approxi- mation technique proposed as an alternative to La- tent Semantic Analysis (LSA, Landauer and Du- mais, 1997). Random Indexing is more scalable and allows for the incremental learning of context information. Curran and Moens (2002) found that dramati- cally increasing the volume of raw input data for distributional similarity tasks increases the accu- racy of synonyms extracted. Random Indexing performs poorly on these volumes of data. Noting that in many NLP tasks, including distributional similarity, statistical weighting is used to improve performance, we modify the Random Indexing al- gorithm to allow for weighted contexts. We test the performance of the original and our modified system using existing evaluation metrics. We further evaluate against bilingual lexicon ex- traction using distributional similarity …

Content truncated to 500 words