Organizing Encyclopedic Knowledge based on the Web and its Application to Question Answering

Organizing Encyclopedic Knowledge based on the Web and its Application to Question Answering Atsushi Fujii University of Library and Information Science 1-2 Kasuga, Tsukuba 305-8550, Japan CREST, Japan Science and Technology Corporation fujii@ulis.ac.jp Tetsuya Ishikawa University of Library and Information Science 1-2 Kasuga, Tsukuba 305-8550, Japan ishikawa@ulis.ac.jp Abstract We propose a method to generate large-scale encyclopedic knowledge, which is valuable for much NLP research, based on the Web. We first search the Web for pages contain- ing a term in question. Then we use lin- guistic patterns and HTML structures to ex- tract text fragments describing the term. Fi- nally, we organize extracted term descrip- tions based on word senses and domains. In addition, we apply an automatically gener- ated encyclopedia to a question answering system targeting the Japanese Information- Technology Engineers Examination. 1 Introduction Reflecting the growth in utilization of the World Wide Web, a number of Web-based language processing methods have been proposed within the natural lan- guage processing (NLP), information retrieval (IR) and artificial intelligence (AI) communities. A sam- ple of these includes methods to extract linguistic resources (Fujii and Ishikawa, 2000; Resnik, 1999; Soderland, 1997), retrieve useful information in re- sponse to user queries (Etzioni, 1997; McCallum et al., 1999) and mine/discover knowledge latent in the Web (Inokuchi et al., 1999). In this paper, mainly from an NLP point of view, we explore a method to produce linguistic resources. Specifically, we enhance the method proposed by Fu- jii and Ishikawa (2000), which extracts encyclopedic knowledge (i.e., term descriptions) from the Web. In brief, their method searches the Web for pages containing a term in question, and uses linguistic ex- pressions and HTML layouts to extract fragments de- scribing the term. They also use a language model to discard non-linguistic fragments. In addition, a clus- tering method is used to divide descriptions into a spe- cific number of groups. On the one hand, their method is expected to en- hance existing encyclopedias, where vocabulary size is relatively limited, and therefore the quantity prob- lems has been resolved. On the other hand, encyclopedias extracted from the Web are not comparable with existing ones in terms of quality. In hand-crafted encyclopedias, term descrip- tions are carefully organized based on domains and word senses, which are especially effective for human usage. However, the output of Fujii’s method is simply a set of unorganized term descriptions. Although clus- tering is optionally performed, resultant clusters are not necessarily related to explicit criteria, such as word senses and domains. To sum up, our belief is that by combining extrac- tion and organization methods, we can enhance both quantity and quality of Web-based encyclopedias. Motivated by this background, we introduce an or- ganization model to Fujii’s method and reformalize the whole framework. In other words, our proposed method is not only extraction but generation of ency- clopedic knowledge. Section 2 explains the overall design of our ency- clopedia generation system, and Section 3 elaborates on our organization model. Section 4 then explores a method …

Content truncated to 500 words