A New Measure for Extracting Semantically Related Words

A New Measure for Extracting Semantically Related Words Yuanyong Wang and Achim Hoffmann wyy@cse.unsw.edu.au achim@cse.unsw.edu.au School of Computer Science and Engineering University of New South Wales Sydney, 2052, NSW Australia Abstract The identification of semantically related terms for a given word is an important problem. A number of statistical approaches have been proposed to ad- dress this problem. Most approaches draw their statistics from a large general corpus. In this pa- per, we propose to use specialized corpora which focus strongly on the individual words of interest. We propose to collect such corpora through targeted queries to Internet search engines. Furthermore, we introduce a new statistical measure, Relative Fre- quency Ratio,tailored specifically for such special- ized corpora. We evaluated our approach by using the extracted related terms to attack the target word selection problem in machine translation. This type of indirect evaluation is conducted because a direct evaluation on the set of related terms thus extracted relies heavily on direct human involvement and is not quantitatively comparable to others’ results. Our experimental results so far are very encouraging. 1 Introduction The identification of semantically related words from texts is an important problem in natural lan- guage processing. If successfully identified, they could be used in query expansion, word sense disambiguation, as well as document classifica- tion (Tomohiko Sugimachi and Matsuo, 2003). Another application concerns the identification of new word senses in specialized languages, which are constantly evolving and, hence, no up-to-date dictionaries exist that could cover all those word senses. Many approaches, such as co-occurrence statistics based on mutual information (Church and Hanks, 1990), the Z-score (Tomohiko Sugimachi and Matsuo, 2003), have been used in the past to tackle this problem. These approaches are limited to be used only on general corpora in which large amounts of texts are collected from sources as di- verse as possible. In this paper, we call this kind of corpus a General Corpus (GC). The nature of these measures (rate high co-occurrence high and rate high frequency words low) requires generality of the corpus. Generality is defined in our paper as for a corpus not being biased toward any particular domain or particular word. Mutual information is defined as follows: I(w1, w2) = log P(w1, w2) P(w1)P(w2), where P(wi) is the probability of word wi to occur in a document and P(wi, wj) is the probability of both words wi and wj to occur in a document. If a good corpus with high generality could be obtained, the co-occurrence statistics collected from the corpus could be very good and reflect accurately the tendency of semantic association among words. But there are several innate drawbacks to these GC based approaches. Firstly, generality is often very difficult to define. Secondly, acquiring a GC with good generality is a even more difficult problem. Thirdly, given that these above two problems are properly addressed the set of related terms extracted from the corpora are still limited in number. The main reason behind the third drawback is actually the …

Content truncated to 500 words