The Importance of Proper Weighting Methods

The Importance of Proper Weighting Methods Chris Buckley Department of Computer Science Cornell University Ithaca, NY 14853 ABSTRACT The importance of good weighting methods in information retrieval -- methods that stress the most useful features of a document or query representative -- is examined. Evidence is presented that good weighting methods are more important than the feature selection process and it is suggested that the two need to go hand-in-hand in order to be effective. The paper concludes with a method for learning a good weight for a term based upon the characteristics of that term. 1. INTRODUCTION Other than experimental results, the first part of this paper contains little new material. Instead, it's an at- tempt to demonstrate the relative importance and dif- ficulties involved in the common information retrieval task of forming documents and query representatives and weighting features. This is the sort of thing that tends to get passed by word of mouth if at all, and never gets published. However, there is a tremendous revival of in- terest in information retrieval; thus this attempt to help all those new people just starting in experimental infor- mation retrieval. A common approach in many areas of natural language processing is to 1. Find "features" of a natural language excerpt 2. Determine the relative importance of those features within the excerpt 3. Submit the weighted features to some task- appropriate decision procedure This presentation focuses on the second sub:task above: the process of weighting features of a natural language representation. Features here could be things like sin- gle word occurrences, phrase occurrences, other relation- ships between words, occurrence of a word in a title, part-of-speech of a word, automatically or manually as- signed categories of a document, citations of a document, and so on. The particular overall task addressed here is that of information retrieval - finding textual documents (from a large set of documents) that are relevant to a user's information need. Weighting features is something that many information retrieval systems seem to regard as being of minor importance as compared fo finding the features in the first place; but the experiments described here suggest that weighting is considerably more impor- tant than additional feature selection. This is not an argument that feature selection is unim- portant, but that development of feature selection and methods of weighting those features need to proceed hand-in-hand if there is to be hope of improving perfor- mance. There have been many papers (and innumerable unpublished negative result experiments) where authors have devoted tremendous resources and intellectual in- sights into finding good features to help represent a doc- ument, but then weighted those features in a haphaz- ard fashion and ended up with little or no improvement. This makes it extremely difficult for a reader to judge the worthiness of a feature approach, especially since the weighting methods are very often not described in detail. Long term, the best weighting methods will obviously be those that can adapt weights as more information be- …

Content truncated to 500 words