A Comparison of Document, Sentence, and Term Event Spaces

Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the ACL, pages 601–608, Sydney, July 2006. c⃝2006 Association for Computational Linguistics A Comparison of Document, Sentence, and Term Event Spaces Catherine Blake School of Information and Library Science University of North Carolina at Chapel Hill North Carolina, NC 27599-3360 cablake@email.unc.edu Abstract The trend in information retrieval sys- tems is from document to sub-document retrieval, such as sentences in a summari- zation system and words or phrases in question-answering system. Despite this trend, systems continue to model lan- guage at a document level using the in- verse document frequency (IDF). In this paper, we compare and contrast IDF with inverse sentence frequency (ISF) and in- verse term frequency (ITF). A direct comparison reveals that all language models are highly correlated; however, the average ISF and ITF values are 5.5 and 10.4 higher than IDF. All language models appeared to follow a power law distribution with a slope coefficient of 1.6 for documents and 1.7 for sentences and terms. We conclude with an analysis of IDF stability with respect to random, journal, and section partitions of the 100,830 full-text scientific articles in our experimental corpus. 1 Introduction The vector based information retrieval model identifies relevant documents by comparing query terms with terms from a document corpus. The most common corpus weighting scheme is the term frequency (TF) x inverse document fre- quency (IDF), where TF is the number of times a term appears in a document, and IDF reflects the distribution of terms within the corpus (Salton and Buckley, 1988). Ideally, the system should assign the highest weights to terms with the most discriminative power. One component of the corpus weight is the language model used. The most common lan- guage model is the Inverse Document Fre- quency (IDF), which considers the distribution of terms between documents (see equation (1)). IDF has played a central role in retrieval systems since it was first introduced more than thirty years ago (Sparck Jones, 1972). IDF(ti)=log2(N)–log2(ni)+1 (1) N is the total number of corpus documents; ni is the number of docu- ments that contain at least one oc- currence of the term ti; and ti is a term, which is typically stemmed. Although information retrieval systems are trending from document to sub-document re- trieval, such as sentences for summarization and words, or phrases for question answering, sys- tems continue to calculate corpus weights on a language model of documents. Logic suggests that if a system identifies sentences rather than documents, it should use a corpus weighting scheme based on the number of sentences rather than the number documents. That is, the system should replace IDF with the Inverse Sentence Frequency (ISF), where N in (1) is the total number of sentences and ni is the number of sen- tences with term i. Similarly, if the system re- trieves terms or phrases then IDF should be re- placed with the Inverse Term Frequency (ITF), where N in (1) is the vocabulary …

Content truncated to 500 words