Query Ambiguity Revisited: Clickthrough Measures for Distinguishing Informational and Ambiguous Queries

Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the ACL, pages 361–364, Los Angeles, California, June 2010. c⃝2010 Association for Computational Linguistics Query Ambiguity Revisited: Clickthrough Measures for Distinguishing Informational and Ambiguous Queries Yu Wang Math & Computer Science Department Emory University yu.wang@emory.edu Eugene Agichtein Math & Computer Science Department Emory University eugene@mathcs.emory.edu Abstract Understanding query ambiguity in web search remains an important open problem. In this paper we reexamine query ambiguity by ana- lyzing the result clickthrough data. Previously proposed clickthrough-based metrics of query ambiguity tend to conflate informational and ambiguous queries. To distinguish between these query classes, we introduce novel met- rics based on the entropy of the click distri- butions of individual searchers. Our exper- iments over a clickthrough log of commer- cial search engine demonstrate the benefits of our approach for distinguishing informational from truly ambiguous queries. 1 Introduction Since query interpretation is the first crucial step in the operation of the web search engines, more re- liable query intent classification, such as detecting whether a query is ambiguous, could allow a search engine to provide more diverse results, better query suggestions, or otherwise improve user experience. In this paper we re-examine query ambiguity in connection with searcher clickthrough behavior. That is, we posit that clickthrough information could provide important evidence for classifying query ambiguity. However, we find that previously pro- posed clickthrough-based measures tend to conflate informational and ambiguous queries. We propose a novel clickthrough measure for query classification, user click entropy, and show that it helps distinguish between informational and truly ambiguous queries. Previous research on this topic focused on binary classification of query ambiguity. Notably, (Tee- van et al., 2008) used click entropy as a proxy for query ambiguity to estimate the potential for search personalization. (Mei and Church, 2008) considered click entropy as measure of search difficulty. More broadly, clickthrough information has been used for many other tasks such as improving search rank- ing (Zhu and Mishne, 2009), learning semantic cat- egories (Komachi et al., 2009), and for topical query classification (Li et al., 2008). However, our work sheds new light on distinguishing between informa- tional and ambiguous queries, by using clickthrough data. Our contributions include: • More precise definition of query ambiguity in terms of clickthrough behavior (Section 2). • Entropy-based formalization of resulting click be- haviors (Section 3). • Empirical validation of our methods on a large real query and clickthrough log (Section 4). 2 Defining Query Ambiguity In this study we focus on two orthogonal query in- tent dimensions, adapted from the top level of user goal taxonomies such as (Rose and Levinson, 2004). Specifically, a query could be ambiguous or unam- biguous; as well as informational or navigational. Consider the example queries of each type below: Ambiguous Unambiguous Informational “al pacino” “lyrics” Navigational “people” “google” The query “al pacino”, the name of a famous ac- tor, is a typical ambiguous and informational query. In the clickthrough logs that we examined, the most popular …

Content truncated to 500 words