Choosing a Distance Metric for Automatic Word Categorization

/ / / / / / / / / CHOOSING A DISTANCE METRIC FOR AUTOMATIC WORD CATEGORIZATION Emin Erkan Korkmaz G6ktiirk U~;oluk Department of Computer Engineering Middle East Technical University Ankara-Turkey Emails: korkmaz~ceng.metu.edu.tr ucolu k@ceng.metu.ed u.tr Abstract This paper analyzes the functionality of dif- ferent distance metrics that can be used in a bottom-up unsupervised algorithm for au- tomatic word categorization. The proposed method uses a modified greedy-type algorithm. The formulations of fuzzy theory are also used to calculate the degree of membership for the elements in the linguistic clusters formed. The unigram and the bigram statistics of a corpus of about two million words are used. Empiri- cal comparisons are made in order to support the discussions proposed for the type of dis- tance metric that would be most suitable for measuring the similarity between linguistic el- ements. 1 Introduction Statistical natural language processing is a challeng- ing area in the field of computational natural lan- guage learning. Researchers of this field have an approach to language acquisition in which learning is visualized as developing a generative, stochastic model of language and putting this model into prac- tice (Marcken, 1996). Automatic word categorization is an important field of application in statistical natural language processing where the process is unsupervised and is carried out by working on n-gram statistics to find out the categories of words. Research in this area points out that it is possible to determine the struc- ture of a natural language by examining the regu- laxities of the statistics of language (Finch, 1993). It is possible to construct a bottom-up unsuper- vised algorithm for the categorization process. In our paper named "A Method for Improving Au- tomatic Word Categorization"(Korkmaz&Uqoluk, 1997) such a method, using a modified greedy-type algorithm supported by the notions of fuzzy logic, has been proposed. The distance metric used to measure the similarities of linguistic elements in this research is the Manhattan Metric. This metric is based on the absolute difference between the corre- sponding values of vector components. The compo- nents of the vectors correspond to bigrarn statistics of words for our case. However words from the same linguistic category in natural language may have to- tally different frequencies. So using a distance met- ric based on only the absolute differences may not be so suitable for the linguistic categorization process. In this paper various distance metrics are analyzed with the same algorithm in order to find out the most suitable one that could be used for linguistic elements. Comparisons are made for the results ob- tained using different metrics. The organization of this paper is as follows. First the related work in the area of word categorization is presented in section 2. Then a general descrip- tion of the categorization process and our proposed algorithm is given in 3 section, which is followed by presentation of different distance metrics that can be used with the algorithm. In section 5 the results of the experiments and the comparisons between the metrics are …

Content truncated to 500 words