Discovery of Term Variation in Japanese Web Search Queries

Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 1484–1492, Singapore, 6-7 August 2009. c⃝2009 ACL and AFNLP Discovery of Term Variation in Japanese Web Search Queries Hisami Suzuki, Xiao Li, and Jianfeng Gao Microsoft Research, Redmond One Microsoft Way, Redmond, WA 98052 USA {hisamis,xiaol,jfgao}@microsoft.com Abstract In this paper we address the problem of identi- fying a broad range of term variations in Japa- nese web search queries, where these varia- tions pose a particularly thorny problem due to the multiple character types employed in its writing system. Our method extends the tech- niques proposed for English spelling correc- tion of web queries to handle a wider range of term variants including spelling mistakes, va- lid alternative spellings using multiple charac- ter types, transliterations and abbreviations. The core of our method is a statistical model built on the MART algorithm (Friedman, 2001). We show that both string and semantic similarity features contribute to identifying term variation in web search queries; specifi- cally, the semantic similarity features used in our system are learned by mining user session and click-through logs, and are useful not only as model features but also in generating term variation candidates efficiently. The proposed method achieves 70% precision on the term variation identification task with the recall slightly higher than 60%, reducing the error rate of a naïve baseline by 38%. 1 Introduction Identification of term variations is fundamental to many NLP applications: words (or more gen- erally, terms) are the building blocks of NLP ap- plications, and any robust application must be able to handle variations in the surface represen- tation of terms, be it a spelling mistake, valid spelling variation, or abbreviation. In search ap- plications, term variations can be used for query expansion, which generates additional query terms for better matching with the terms in the document set. Identifying term variations is also useful in other scenarios where semantic equiva- lence of terms is sought, as it represents a very special case of paraphrase. This paper addresses the problem of identify- ing term variations in Japanese, specifically for the purpose of query expansion in web search, which appends additional terms to the original query string for better retrieval quality. Query expansion has been shown to be effective in im- proving web search results in English, where dif- ferent methods of generating the expansion terms have been attempted, including relevance feed- back (e.g., Salton and Buckley, 1990), correction of spelling errors (e.g., Cucerzan and Brill, 2004), stemming or lemmatization (e.g., Frakes, 1992), use of manually- (e.g., Aitchison and Gilchrist, 1987) or automatically- (e.g., Rasmussen 1992) constructed thesauri, and Latent Semantic Index- ing (e.g., Deerwester et al, 1990). Though many of these methods can be applied to Japanese query expansion, there are unique problems posed by Japanese search queries, the most chal- lenging of which is that valid alternative spel- lings of a word are extremely common due to the multiple script types employed in the language. For example, the word for …

Content truncated to 500 words