Combining Multiple Forms of Evidence While Filtering

Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing (HLT/EMNLP), pages 587–595, Vancouver, October 2005. c⃝2005 Association for Computational Linguistics Combining Multiple Forms of Evidence While Filtering Yi Zhang ∗ Information System and Technology Management School of Engineering University of California, Santa Cruz Santa Cruz, CA 95064, USA yiz@soe.ucsc.edu Jamie Callan Language Technologies Institute School of Computer Science Carnegie Mellon University Pittsburgh, PA 15213, USA callan@cs.cmu.edu Abstract This paper studies how to go beyond relevance and enable a filtering system to learn more in- teresting and detailed data driven user models from multiple forms of evidence. We carry out a user study using a real time web based per- sonal news filtering system, and collect exten- sive multiple forms of evidence, including ex- plicit and implicit user feedback. We explore the graphical modeling approach to combine these forms of evidence. To test whether the ap- proach can help us understand the domain bet- ter, we use graph structure learning algorithm to derive the causal relationships between dif- ferent forms of evidence. To test whether the approach can help the system improve the per- formance, we use the graphical inference algo- rithms to predict whether a user likes a docu- ment based on multiple forms of evidence. The results show that combining multiple forms of evidence using graphical models can help us better understand the filtering problem, im- prove filtering system performance, and handle various data missing situations naturally. 1 Introduction An adaptive personal information filtering system is an autonomous agent that delivers information to the user in a dynamic environment over a period of time. A com- mon filtering approach is adapting existing text classi- fication/retrieval algorithms to classify incoming docu- ments as either relevant or non relevant using user pro- files learned from explicit user feedback on documents the user has seen. However, there are other important criteria for the user besides relevance, such as readabil- ity (Collins-Thompson and Callan, 2004), novelty (Har- man, 2003), and authority (Kleinberg, 1998). Besides, much information about the user and the document can be collected by a filtering system. These suggest a way to improve the current filtering system: going beyond rele- vance and using multiple forms of evidence. ∗This research was done while at the Language Technolo- gies Institute, Carnegie Mellon University. Unfortunately, there is no standard evaluation data set for this research, and there is not much work on finding a good theory to combine various forms of evidence. To solve the first problem, we designed a user study and col- lect thousands of cases with multiple forms of evidence, including the content of a document, explicit and im- plicit user feedback, such as a user’s mouse usage, key board usage, document length, novelty, relevance, read- ability, authority, user profile characteristics, news source information, and whether a user likes a document or not. Solving the second problem is very challenging. A good model should have the representation power to combine multiple forms of evidence; it …

Content truncated to 500 words