TSUBAKI: An Open Search Engine Infrastructure for Developing New Information Access Methodology

TSUBAKI: An Open Search Engine Infrastructure for Developing New Information Access Methodology Keiji Shinzato†, Tomohide Shibata†, Daisuke Kawahara‡, Chikara Hashimoto‡‡ and Sadao Kurohashi† †Graduate School of Informatics, Kyoto University ‡National Institute of Information and Communications Technology ‡‡Department of Informatics, Yamagata University {shinzato, shibata, kuro}@nlp.kuee.kyoto-u.ac.jp dk@nict.go.jp ch@yz.yamagata-u.ac.jp Abstract As the amount of information created by human beings is explosively grown in the last decade, it is getting extremely harder to obtain necessary information by conven- tional information access methods. Hence, creation of drastically new technology is needed. For developing such new technol- ogy, search engine infrastructures are re- quired. Although the existing search engine APIs can be regarded as such infrastructures, these APIs have several restrictions such as a limit on the number of API calls. To help the development of new technology, we are run- ning an open search engine infrastructure, TSUBAKI, on a high-performance comput- ing environment. In this paper, we describe TSUBAKI infrastructure. 1 Introduction As the amount of information created by human be- ings is explosively grown in the last decade (Uni- versity of California, 2003), it is getting extremely harder to obtain necessary information by con- ventional information access methods, i.e., Web search engines. This is obvious from the fact that knowledge workers now spend about 30% of their day on only searching for information (The Del- phi Group White Paper, 2001). Hence, creation of drastically new technology is needed by integrating several disciplines such as natural language process- ing (NLP), information retrieval (IR) and others. Conventional search engines such as Google and Yahoo! are insufficient to search necessary informa- tion from the current Web. The problems of the con- ventional search engines are summarized as follows: Cannot accept queries by natural language sen- tences: Search engine users have to represent their needs by a list of words. This means that search engine users cannot obtain necessary information if they fail to represent their needs into a proper word list. This is a serious problem for users who do not utilize a search engine frequently. Cannot provide organized search results: search result is a simple list consisting of URLs,A titles and snippets of web pages. This type of re- sult presentation is obviously insufficient consider- ing explosive growth and diversity of web pages. Cannot handle synonymous expressions: Exist- ing search engines ignore a synonymous expression problem. Especially, since Japanese uses three kinds of alphabets, Hiragana, Katakana and Kanji, this problem is more serious. For instance, although both Japanese words “こども” and “子供” mean child, the search engines provide quite different search re- sults for each word. We believe that new IR systems that overcome the above problems give us more flexible and com- fortable information access and that development of such systems is an important and interesting re- search topic. To develop such IR systems, a search engine in- frastructure that plays a low-level layer role (i.e., re- trieving web pages according to a user’s query from a huge web page collection) is required. The Appli- …

Content truncated to 500 words