Cumulative Progress in Language Models for Information Retrieval

Cumulative Progress in Language Models for Information Retrieval Antti Puurula The University of Waikato Private Bag 3105 Hamilton 3240, New Zealand asp12@students.waikato.ac.nz Abstract The improvements to ad-hoc IR sys- tems over the last decades have been re- cently criticized as illusionary and based on incorrect baseline comparisons. In this paper several improvements to the LM approach to IR are combined and evaluated: Pitman-Yor Process smooth- ing, TF-IDF feature weighting and model- based feedback. The increases in rank- ing quality are significant and cumulative over the standard baselines of Dirichlet Prior and 2-stage Smoothing, when eval- uated across 13 standard ad-hoc retrieval datasets. The combination of the improve- ments is shown to improve the Mean Av- erage Precision over the datasets by 17.1% relative. Furthermore, the considered im- provements can be easily implemented with little additional computation to exist- ing LM retrieval systems. On the basis of the results it is suggested that LM research for IR should move towards using stronger baseline models. 1 Introduction Research on ad-hoc Information Retrieval (IR) has been recently criticized for being based on incorrect baseline comparisons. According to extensive evaluation of IR systems from over a decade, no progress has been demonstrated on standard datasets (Armstrong et al., 2009a; Armstrong et al., 2009b). In this paper we propose that although much of this criticism is valid, much of the more recent progress in Language Model-based (LM) IR has not been evaluated or received the attention that it deserved. We evaluate on 13 standard IR datasets some of the improvements that have been suggested to LMs over the years. It is shown that the combination of Pitman-Yor Process smooth- ing, TF-IDF feature weighting and Model-based Feedback produces a substantial and cumulative improvement over the common baseline LM smoothing methods. 2 Improvements to LMs for IR 2.1 LM Approach to IR The LM approach to ad-hoc IR considers docu- ments and queries to be generated by underlying n-gram LMs. The Query Likelihood (QL) frame- work for LM retrieval (Hiemstra, 1998) treats queries as being generated by document models, reducing the retrieval of the most relevant doc- uments into ranking documents by the posterior probability of each document given the query. Un- igram LMs and a uniform distribution over docu- ment priors is commonly assumed, so that the QL- score for each document correspond to the condi- tional log-probability of the query given the docu- ment: log pm(w) = log Z(w) + � n wn log pm(n), (1) where Z(w) is a Multinomial normalizer, w is the query word count vector, and pm(n) is given by a Multinomial estimated from the document word count vector dm: pm(n) = dmn ||dm||1 (2) The QL framework is the standard application of LMs to IR. It is equivalent to using a Multino- mial Naive Bayes model for ranking, with classes corresponding to documents, and a uniform prior over the document models. Antti Puurula. 2013. Cumulative Progress in Language Models for Information Retrieval. In Proceedings of Australasian Language Technology Association Workshop, …

Content truncated to 500 words