Learning a Replacement Model for Query Segmentation with Consistency in Search Logs

International Joint Conference on Natural Language Processing, pages 28–36, Nagoya, Japan, 14-18 October 2013. Learning a Replacement Model for Query Segmentation With Consistency in Search Logs Wei Zhang§∗, Yunbo Cao‡, Chin-Yew Lin‡, Jian Su§, Chew-Lim Tan† §Institute for Infocomm Research, ‡Microsoft Research Asia †National University of Singapore {zhangw3,sujian}@i2r.a-star.edu.sg {yunbo.cao,cyl}@microsoft.com, tancl@comp.nus.edu.sg Abstract Query segmentation is to split a query into a sequence of non-overlapping segments that completely cover all tokens in the query. The majority of methods are unsupervised, how- ever, they are usually not as accurate as su- pervised methods due to the lack of guidance from labeled data. In this paper, we propose a new paradigm of learning a replacement model with consistency (LRMC), to enable un- supervised training with guidance from search log data. In LRMC, we first assume the ex- istence of a base segmenter (an implementa- tion of any existing approach). Then, we uti- lize a key observation that queries with a sim- ilar intent tend to have consistent segmenta- tions, to automatically collect a set of labeled data from the outputs of the base segmenter by leveraging search log data. Finally, we employ the auto-collected data to train a replacement model for selecting the correct segmentation of a new query from the outputs of the base segmenter. The results show LRMC can im- prove state-of-the-art methods by an F-Score of around 7%. 1 Introduction Nowadays keyword queries have been adopted as the de-facto query interface by most search en- gines. Query tokens are not independent or un- ordered symbols but rather ordered and structured words and phrases with syntactic relationships. Understanding the structure of a query is crucial for achieving better search performance. Such an understanding will also ease other search re- lated applications such as query suggestion and rewriting, where one is able to work on seman- tic concepts instead of individual tokens. Query segmentation (QS), a process of splitting a query into a sequence of non-overlapping segments that ∗Wei Zhang did this work when he was an intern at Mi- crosoft Research Asia. completely cover all tokens, aims to address these challenges. It requires that every segment ren- dered is a phrase or a semantic unit. For exam- ple, given a query “download adobe writer”, four different ways of segmentation are possible. The challenge is to determine which one is correct. The majority of QS methods are unsupervised, however, they are not as accurate as supervised methods due to lack of guidance from labeled data. On the other hand, supervised models suf- fer from the problems: (1) new phrases/words are introduced on the web daily, which quickly inval- idate static supervised models trained on a certain manually labeled set; (2) it is not feasible to de- velop a set of labeled data covering all domains on the web. In this paper, we propose a paradigm of learning a replacement model with consistency (LRMC), to enable unsupervised training and it improves various unsupervised QS systems. LRMC first assumes the existence of a base seg- …

Content truncated to 500 words