Factual or Satisfactory: What Search Results Are Better?

Factual or Satisfactory: What Search Results Are Better?1 Yu Hong, Jun Lu, Jianmin Yao, Guodong Zhou, Qiaoming Zhu aSchool of Computer Science and Technology, Soochow University, No.1 Shizi Street, Suzhou City and 215006, China {hongy, 20104227066, jyao, gdzhou, qmzhu}@suda.edu.cn Abstract. An interesting issue “whether the factual search result is better than the satisfactory one?” is discussed in this paper, and all effort is to illustrate the publicly recognized conception “the supreme satisfactory result is optimal” could lead Information Retrieval astray. By contrast, we propose a new hypothesis “the factually result should be the optimal although sometimes unsatisfactory”. To implement the pilot study on this, we developed a search-engine (Google) based labeling platform and designed a new evaluation method involving user experiences. The results on the platform show the existence of bias on determining the optimal search results (factual or satisfactory). And the very different performances of NDCG and our evaluation metric demonstrate the weakness of current ranking algorithms in mining and recommending the long-term effective information. Keywords: Factual search result, satisfactory search result, information retrieval, evaluation 1 Introduction What services are current search engine providing? The universal search engines are mining and recommending all possible relevant information to users. Further, the personal search engines are learning the real intention of users and recommending the information that extremely satisfies the personal requirements. But whether are the satisfactory search results factually factual? The answer should be no. In detail, it can be sure that users aim to explore the unknown when using the search engine, and thus they should have little or even no prior knowledge to support their judgments on the satisfaction with the search results. This may result in the probability that any result with a little relation to the query (even wrong one) can be determined to be satisfactory. It just like that a lost visitor would always say “thank you” to the helpers even if he was shown the wrong way. Both the explicit and the implicit feedbacks can reflect the user experience of search results (satisfied or not). So the feedbacks are normally used to adaptively learn user intention and iteratively improve the retrieval performance. However, as the discussion above, the judgment on the satisfaction might be imprecise. Thus the adaptive learning, as the most important component of personal search engines, is possibly using the unreasonable feedbacks to mislead the retrieval process. The doubts mentions above raise two questions: 1) Whether are the satisfactory results always the helpful information for users to acquire knowledge? And whether can they consistently enhance the intelligence of information retrieval? 2) If the answer is no, whether does there exist other kind of information to compensate for the potential failings of users even or supersede the satisfactoriness? In this paper, we propose a hypothesis that the better search results should be the factual information. The factualness mentioned here is not “users think something right or find it nice” but means the nature of information is really right. The best factual information should …

Content truncated to 500 words