Semantic-based Estimation of Term Informativeness

Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the ACL, pages 530–538, Boulder, Colorado, June 2009. c⃝2009 Association for Computational Linguistics Semantic-based Estimation of Term Informativeness Kirill Kireyev University of Colorado – Boulder kireyev@colorado.edu Abstract The idea that some words carry more semantic content than others, has led to the notion of term specificity, or informativeness. Computa- tional estimation of this quantity is important for various applications such as information re- trieval. We propose a new method of comput- ing term specificity, based on modeling the rate of learning of word meaning in Latent Semantic Analysis (LSA). We analyze the performance of this method both qualitatively and quantitat- ively and demonstrate that it shows excellent performance compared to existing methods on a broad range of tests. We also demonstrate how it can be used to improve existing applica- tions in information retrieval and summariza- tion. 1 Introduction The idea that some words carry more semantic content than others has been occurring in various literature in linguistics, psychology and computer science for some time. The intuitive notion of spe- cificity has long existed before it was formalized; consider, for example, the distinction between the more general word “beverage” and more specific terms “tea”, “coffee” and “cocoa” made by Spär- ck-Jones (1973). Another informal mention of spe- cificity is mentioned by Gorman (1961): A word may be “abstract” and either general or specific, or “concrete” and either general or specific. where it is contrasted with another psycholinguistic property of concreteness, which is generally defined as “the extent to which the word's referent can be touched or felt” (Reilly et al., 2007). The field of information retrieval has attracted greater attention to the computational estimation and applications of term specificity. It has been noted that words with higher specificity, or inform- ation content, deserve to be weighted more heavily when matching documents with queries, since these words play a greater importance in character- izing what a query or a document is about. By con- trast, stopwords, words that contribute the least amount of semantic content, are often down- weighted or removed altogether (see (Lo et al., 2005), for example). In addition to IR, term specificity, or informat- iveness, has been shown useful in other applica- tions, such as Named Entity Tagging (Rennie et al., 2005), creating back-of-the-book glossaries (Csomai et al., 2007), and extractive summariza- tion (Kireyev, 2008). A related notion of communication density has been introduced by Gorman et al. (2003) in team communication analysis, to measure the extent to which a team conveys information in a concise manner, or, in other words, the rate of meaningful discourse, defined by the ratio of meaningfulness to number of words spoken. The meaningfulness described here should not be confused with psy- cholinguistic quality of meaningfulness as de- scribed by Toglia and Battig (1978), which is the degree to which a word is associated with other words. In this paper we consider the terms specificity, informativeness and information content of …

Content truncated to 500 words