Evaluation of Automatically Identified Index Terms for Browsing Electronic Documents

Evaluation of Automatically Identified Index Terms for Browsing Electronic Documents I Nina Wacholder, Judith L. Klavans and David K. Evans Columbia University Department of Computer Science and Center for Research on Information Access 1. Abstract We present an evaluation of domain- independent natural language tools for use in the identification of significant concepts in documents. Using qualitative evaluation, we compare three shallow processing methods for extracting index terms, i.e., terms that can be used to model the content of documents. We focus on two criteria: quality and coverage. In terms of quality alone, our results show that technical term (TT) extraction [Justeson and Katz 1995] receives the highest rating. How- ever, in terms of a combined quality and cover- age metric, the Head Sorting (HS) method, described in [Wacholder 1998], outperforms both other methods, keyword (KW) and TT. 2. Introduction In this paper, we consider the problem of how to evaluate the automatic identification of index terms that have been derived without recourse to lexicons or to other kinds of domain-specific information. By index terms, we mean natural language expressions that constitute a meaning- ful representation of a document for humans. The premise of this research is that if signifi- cant topics coherently represent information in a document, these topics can be used as index terms that approximate the content of individ- ual documents in large collections of electronic documents. We compare three shallow processing • methods for identifying index terms: Keywords (KW) are terms identified by counting frequency of stemmed words in a document; Technical terms (TT) are noun phrases (NPs) or subparts of NPs repeated more than twice in a document [Justeson and Katz 1995]; Head sorted terms (HS) are identified by a method in which simplex noun phrases (as defined below) are sorted by head and then ranked in decreasing order of fre- quency [Wacholder 1998]. The three methods that we evaluated are do- main-independent in that they use statistical and/or linguistic properties that apply to any natural language document in any field. These methods are also corpus-independent, in that the ranking of terms for an individual document is not dependent on properties of the corpus. 2.1 Overview of methods and results Subjects were drawn from two groups: professionals and students. Professionals in- cluded librarians and publishing professionals familiar with both manual and automatic text indexing. Students included undergraduate and graduate students with a variety of academic interests. To assess terms, we used a standard qualitative ranking technique. We presented subjects with an article and a list of terms identified by one of the three methods. Subjects were asked to answer the following general question: "Would this term be useful in an electronic index for this article?" Terms were rated on a scale of 1 to 5, where 1 indicates a high quality term that should definitely be in- cluded in the index and 5 indicates a junk term that definitely should not be included. For ex- 1 This research was partly funded by NSF IRI 97-12069, …

Content truncated to 500 words