Empirical Exploitation of Click Data for Task Specific Ranking

Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 1086–1095, Singapore, 6-7 August 2009. c⃝2009 ACL and AFNLP Empirical Exploitation of Click Data for Task Specific Ranking Anlei Dong Yi Chang Shihao Ji Ciya Liao Xin Li Zhaohui Zheng Yahoo! Labs 701 First Avenue Sunnyvale, CA 94089 {anlei,yichang,shihao,ciyaliao,xinli,zhaohui}@yahoo-inc.com Abstract There have been increasing needs for task specific rankings in web search such as rankings for specific query segments like long queries, time-sensitive queries, navi- gational queries, etc; or rankings for spe- cific domains/contents like answers, blogs, news, etc. In the spirit of ”divide-and- conquer”, task specific ranking may have potential advantages over generic ranking since different tasks have task-specific fea- tures, data distributions, as well as feature- grade correlations. A critical problem for the task-specific ranking is training data insufficiency, which may be solved by us- ing the data extracted from click log. This paper empirically studies how to appro- priately exploit click data to improve rank function learning in task-specific ranking. The main contributions are 1) the explo- ration on the utilities of two promising ap- proaches for click pair extraction; 2) the analysis of the role played by the noise information which inevitably appears in click data extraction; 3) the appropriate strategy for combining training data and click data; 4) the comparison of click data which are consistent and inconsistent with baseline function. 1 Introduction Learning-to-rank approaches (Liu, 2008) have been widely applied in commercial search en- gines, in which ranking models are learned using labeled documents. Significant efforts have been made in attempt to learn a generic ranking model which can appropriately rank documents for all queries . However, web users’ query intentions are extremely heterogeneous, which makes it difficult for a generic ranking model to achieve best rank- ing results for all queries. For this reason, there have been increasing needs for task specific rank- ings in web search such as rankings for specific query segments like long queries, time-sensitive queries, navigational queries, etc; or rankings for specific domains/contents like answers, blogs, news, etc. Therefore, a specific ranking task usu- ally correspond to a category of queries; when the search engine determines that a query is be- longing to this category, it will call the ranking function dedicated to this ranking task. The mo- tivation of this divide-and-conquer strategy is that, task specific ranking may have potential advan- tages over generic ranking since different tasks have task-specific features, data distributions, as well as feature-grade correlations. Such a dedicated ranking model can be trained using the labeled data belonging to this query cat- egory (which is called dedicated training data). However, the amount of training data dedicated to a specific ranking task is usually insufficient because human labeling is expensive and time- consuming, not to mention there are multiple rank- ing tasks that need to be taken care of. To deal with the training data insufficiency problem for task-specific ranking, we propose to extract click- through data and incorporate it with dedicated training …

Content truncated to 500 words