Exploring Distributional Similarity Based Models for Query Spelling Correction

Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the ACL, pages 1025–1032, Sydney, July 2006. c⃝2006 Association for Computational Linguistics Exploring Distributional Similarity Based Models for Query Spelling Correction Mu Li Microsoft Research Asia 5F Sigma Center Zhichun Road, Haidian District Beijing, China, 100080 muli@microsoft.com Muhua Zhu School of Information Science and Engineering Northeastern University Shenyang, Liaoning, China, 110004 zhumh@ics.neu.edu.cn Yang Zhang School of Computer Science and Technology Tianjin University Tianjin, China, 300072 yangzhang@tju.edu.cn Ming Zhou Microsoft Research Asia 5F Sigma Center Zhichun Road, Haidian District Beijing, China, 100080 mingzhou@microsoft.com Abstract A query speller is crucial to search en- gine in improving web search relevance. This paper describes novel methods for use of distributional similarity estimated from query logs in learning improved query spelling correction models. The key to our methods is the property of dis- tributional similarity between two terms: it is high between a frequently occurring misspelling and its correction, and low between two irrelevant terms only with similar spellings. We present two models that are able to take advantage of this property. Experimental results demon- strate that the distributional similarity based models can significantly outper- form their baseline systems in the web query spelling correction task. 1 Introduction Investigations into query log data reveal that more than 10% of queries sent to search engines contain misspelled terms (Cucerzan and Brill, 2004). Such statistics indicate that a good query speller is crucial to search engine in improving web search relevance, because there is little op- portunity that a search engine can retrieve many relevant contents with misspelled terms. The problem of designing a spelling correction program for web search queries, however, poses special technical challenges and cannot be well solved by general purpose spelling correction methods. Cucerzan and Brill (2004) discussed in detail specialties and difficulties of a query spell checker, and illustrated why the existing methods could not work for query spelling correction. They also identified that no single evidence, ei- ther a conventional spelling lexicon or term fre- quency in the query logs, can serve as criteria for validate queries. To address these challenges, we concentrate on the problem of learning improved query spell- ing correction model by integrating distributional similarity information automatically derived from query logs. The key contribution of our work is identifying that we can successfully use the evidence of distributional similarity to achieve better spelling correction accuracy. We present two methods that are able to take advan- tage of distributional similarity information. The first method extends a string edit-based error model with confusion probabilities within a gen- erative source channel model. The second method explores the effectiveness of our ap- proach within a discriminative maximum entropy model framework by integrating distributional similarity-based features. Experimental results demonstrate that both methods can significantly outperform their baseline systems in the spelling correction task for web search queries. 1025 The rest of the paper is structured as follows: after a brief overview of the related work in Sec- tion 2, …

Content truncated to 500 words