Lexical Chains as Document Features

Lexical Chains as Document Features Dinakar Jayarajan, Dipti Deodhare Centre for Artificial Intelligence and Robotics, Defence R & D Organisation, Bangalore, INDIA. [dinakarj, dipti]@cair.drdo.in B Ravindran Dept. of CSE, IIT Madras, Chennai, INDIA. ravi@cse.iitm.ac.in Abstract Document clustering and classification is usually done by representing the documents using a bag of words scheme. This scheme ignores many of the linguistic and semantic features contained in text documents. We propose here an alternative representation for documents using Lexical Chains. We compare the performance of the new repre- sentation against the old one on a cluster- ing task. We show that Lexical Chain based features give better results than the Bag of Words based features, while achieving al- most 30% reduction in the dimensionality of the feature vectors resulting in faster execu- tion of the algorithms. 1 Introduction Text data usually contains complex semantic infor- mation which is communicated using a combination of words. Ideally, the representation used should capture and reflect this fact in order to semantically drive the clustering algorithm and obtain better re- sults. The Bag of Words (BoW) (Salton et al., 1975) scheme is a very popular scheme which has been used for representing documents. But, this scheme ignores many of the linguistic and semantic features contained in text documents. This paper explores an alternative representation for documents, using lexical chains, which encodes some of the semantic information contained in the document. This rep- resentation results in improved performance on the clustering tasks and achieves a drastic reduction in the size of the feature space as well. The BoW scheme was originally designed for the Information Retrieval domain (Salton, 1989) where the aim was to ‘index’ the document and not nec- essarily to model the topic distribution. This rep- resentation has since been adopted as the defacto document representation scheme for supervised and unsupervised learning on documents. The BoW scheme represents features as an unordered set of words contained in the document, along with their frequency count. The BoW scheme assumes that the distribution of words in a document reflect the underlying dis- tribution of topics and hence if the documents are grouped on the basis of the similarity of the words contained in them, it will implicitly result in a clus- tering based on topics. This representation, using a simple frequency count alone, does not capture all the underlying information present in the doc- uments. Moreover, it ignores information such as position, relations and co-occurrences among the words. In addition, the feature space formed will be very huge and sparse resulting in time and space costs as well. Lexical Chaining is a technique which seeks to identify and exploit the semantic relatedness of words in a document. It is based on the phe- nomenon of lexical cohesion (Halliday and Hasan, 1976) and works on the premise that semantically related words co-occur close together in a passage more than “just by chance”. Lexical chaining is the process of identifying and grouping such words to- gether to form chains which …

Content truncated to 500 words