Learning Dense Models of Query Similarity from User Click Logs

Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the ACL, pages 474–482, Los Angeles, California, June 2010. c⃝2010 Association for Computational Linguistics Learning Dense Models of Query Similarity from User Click Logs Fabio De Bona∗ Friedrich Miescher Laboratory of the Max Planck Society T¨ubingen, Germany fabio@tuebingen.mpg.de Stefan Riezler Google Research Z¨urich, Switzerland riezler@google.com Keith Hall Google Research Z¨urich, Switzerland kbhall@google.com Massimiliano Ciaramita Google Research Z¨urich, Switzerland massi@google.com Amac¸ Herdaˇgdelen∗ University of Trento Rovereto, Italy amac@herdagdelen.com Maria Holmqvist∗ Linkopings University Linkopings, Sweden marho@ida.liu.se Abstract The goal of this work is to integrate query similarity metrics as features into a dense model that can be trained on large amounts of query log data, in order to rank query rewrites. We propose features that incorpo- rate various notions of syntactic and semantic similarity in a generalized edit distance frame- work. We use the implicit feedback of user clicks on search results as weak labels in train- ing linear ranking models on large data sets. We optimize different ranking objectives in a stochastic gradient descent framework. Our experiments show that a pairwise SVM ranker trained on multipartite rank levels outperforms other pairwise and listwise ranking methods under a variety of evaluation metrics. 1 Introduction Measures of query similarity are used for a wide range of web search applications, including query expansion, query suggestions, or listings of related queries. Several recent approaches deploy user query logs to learn query similarities. One set of ap- proaches focuses on user reformulations of queries that differ only in one phrase, e.g., Jones et al. (2006). Such phrases are then identified as candi- date expansion terms, and filtered by various signals such as co-occurrence in similar sessions, or log- likelihood ratio of original and expansion phrase. Other approaches focus on the relation of queries and search results, either by clustering queries based ∗The work presented in this paper was done while the au- thors were visiting Google Research, Z¨urich. on their search results, e.g., Beeferman and Berger (2000), or by deploying the graph of queries and re- sults to find related queries, e.g., Sahami and Heil- man (2006). The approach closest to ours is that of Jones et al. (2006). Similar to their approach, we create a train- ing set of candidate query rewrites from user query logs, and use it to train learners. While the dataset used in Jones et al. (2006) is in the order of a few thousand query-rewrite pairs, our dataset comprises around 1 billion query-rewrite pairs. Clearly, man- ual labeling of rewrite quality is not feasible for our dataset, and perhaps not even desirable. Instead, our intent is to learn from large amounts of user query log data. Such data permit to learn smooth mod- els because of the effectiveness of large data sets to capture even rare aspects of language, and they also are available as in the wild, i.e., they reflect the ac- tual input-output behaviour that we seek to automate (Halevy et al., 2009). We propose …

Content truncated to 500 words