A Measure of Term Representativeness Based on the Number of Co-occurring Salient Words

A Measure of Term Representativeness Based on the Number of Co-occurring Salient Words Toru Hisamitsu and Yoshiki Niwa Central Research Laboratory, Hitachi, Ltd. Hatoyama, Saitama, 350-0095, Japan {hisamitu, yniwa}@harl.hitachi.co.jp Abstract We propose a novel measure of the representativeness (i.e., indicativeness or topic specificity) of a term in a given corpus. The measure embodies the idea that the distribution of words co-occurring with a representative term should be biased according to the word distribution in the whole corpus. The bias of the word distribution in the co-occurring words is defined as the number of distinct words whose occurrences are saliently biased in the co-occurring words. The saliency of a word is defined by a threshold probability that can be automatically defined using the whole corpus. Comparative evaluation clarified that the measure is clearly superior to conventional measures in finding topic-specific words in the newspaper archives of different sizes. Introduction Measuring the representativeness (i.e., the informativeness or domain specificity) of a term１ is essential to various tasks in natural language processing (NLP) and information retrieval (IR). Such a measure is particularly crucial to automatic dictionary construction and IR interfaces to show a user words indicative of topics in retrievals that often consist of an intractably large number of documents (Niwa et al. 2000). This paper proposes a novel and effective measure of term representativeness that reflects the bias of the words co-occurring with a term. In the following, we focus on extracting topic words from an archive of newspaper articles. In the literature of NLP and IR, there have been a number of studies on term weighting, and these are strongly related to measures of term １ A term is a word or a word sequence. representativeness (see section 1). In this paper we employ the basic idea of the ‘baseline method’ proposed by Hisamitsu (Hisamitsu et al. 2000). The idea is that the distribution of words co-occurring with a representative term should be biased according to the word distribution of the whole corpus. Concretely, for any term T and any measure M for the degree of bias of word occurrences in D(T), a set of words co-occurring with T, according to those of the whole corpus D0, the baseline method defines representativeness of term T by normalizing M(D(T)). In what follows, D0 is an archive of newspaper articles and D(T) is defined as the set of all articles containing T. The normalization of M(D(T)) is done by a function BM, called the baseline function, which estimates the value of M(Drand) using #Drand for any randomly sampled document (in our case, ‘article’) set Drand, where #Drand stands for the total number of words contained in Drand. By dividing M(D(T)) by BM(#D(T)), comparison of M(D(T1)) and M(D(T2)) becomes meaningful even if the frequencies of T1 and T2 are very different. We denote this normalized value by NormM(D(T)). Hisamitsu et al. reported that NormM(D(T)) is very effective in capturing topic-specific words when M(D(T)) is defined as the distance between two word distributions PD(T) and …

Content truncated to 500 words