Japanese Query Alteration Based on Lexical Semantic Similarity

Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the ACL, pages 191–199, Boulder, Colorado, June 2009. c⃝2009 Association for Computational Linguistics Japanese Query Alteration Based on Semantic Similarity Masato Hagiwara Nagoya University Furo-cho, Chikusa-ku Nagoya 464-8603, Japan hagiwara@kl.i.is.nagoya-u.ac.jp Hisami Suzuki Microsoft Research One Microsoft Way Redmond, WA 98052, USA hisamis@microsoft.com Abstract We propose a unified approach to web search query alterations in Japanese that is not lim- ited to particular character types or ortho- graphic similarity between a query and its al- teration candidate. Our model is based on pre- vious work on English query correction, but makes some crucial improvements: (1) we augment the query-candidate list to include orthographically dissimilar but semantically similar pairs; and (2) we use kernel-based lexical semantic similarity to avoid the prob- lem of data sparseness in computing query- candidate similarity. We also propose an ef- ficient method for generating query-candidate pairs for model training and testing. We show that the proposed method achieves about 80% accuracy on the query alteration task, improv- ing over previously proposed methods that use semantic similarity. 1 Introduction Web search query correction is an important prob- lem to solve for robust information retrieval given how pervasive errors are in search queries: it is said that more than 10% of web search queries contain errors (Cucerzan and Brill, 2004). English query correction has been an area of active research in re- cent years, building on previous work on general- purpose spelling correction. However, there has been little investigation of query correction in lan- guages other than English. In this paper, we address the issue of query cor- rection, and more generally, query alteration in Japanese. Japanese poses particular challenges to the query correction task due to its complex writ- ing system, summarized in Fig. 11. There are four 1The figure is somewhat over-simplified as it does not in- clude any word consisting of multiple character types. It also does not include examples of spelling mistakes and variants in word segmentation. Kanji Sp: 慶応～慶應 Abbr: 東京大学～東大 Abbr: 東急東横線～東横線 Hiragana Sp: あかしあ～あかしや Roman Alphabet Sp: Ohno~Oono Sp: center~centre Katakana Sp: スパゲティ～スパゲッティ Abbr: マクドナルド～マック Abbr: ドラゴンクエスト～ドラクエ Sp: Fedex～フェデックス Abbr: MS～マイクロソフト Sp: びん～ビン Syn: こよみ～カレンダー Sp: 乗換～のりかえ Sp: 花瓶～かびん Sp:蛋白～タンパク Syn: 座席～シート Syn: 全日空～ANA Figure 1: Japanese character types and spelling variants main character types, including two types of kana (phonetic alphabet - hiragana and katakana), kanji (ideographic - characters represent meaning) and Roman alphabet; a word can be legitimately spelled in multiple ways, combining any of these character sets. For example, the word for ‘protein’ can be spelled as たんぱくしつ (all in hiragana), タンパク 質 (katakana+kanji), 蛋白質 (all in kanji) or たん白 質 (hiragana+kanji), all pronounced in the same way (tanpakushitsu). Some examples of these spelling variants are shown in Fig. 1 with the prefix Sp: as is observed from the figure, spelling variation occurs within and across different character types. Resolv- ing these variants will be essential not only for in- formation retrieval but practically for all …

Content truncated to 500 words