A Corpus-Based Statistical Approach to Automatic Book Indexing

A Corpus-Based Statistical Approach to Automatic Book Indexing Jyun-Sheng Chang*, Tsung-Yih Tseng, Ying Cheng, Huey-Chyun Chen, Shun-Der Cheng Department of Computer Science National Tsing Hua University Hsinchu, Taiwan 30043, ROC j schang@cs.nthu.edu.tw Sur-Jin Ker Department of Computer Science SooChow University John S. Liu Software Research Office Sampo Research Institute Abstract The paper reports on a new approach to automatic generation of back-of-book indexes for Chinese books. Parsing on the level of complete sentential analysis is avoided because of the inefficiency and unavailability of a Chinese Grammar with enough coverage. Instead, fundamental analysis particular to Chinese text called word segmentation is performed to break up characters into a sequence of lexical units equivalent to words in English. The sequence of words then goes through part-of- speech tagging and noun phrase analysis. All these analyses are done using a corpus-based statistical algorithm. Experimental results have shown satisfactory results. I. Introduction Preparing back-of-book indexes is of vital importance to the publishing industry but is a very labor intensive task. Attempts have been made over the years to automate this procedure for the apparent benefits of cost saving, shorter preparation time, and possibility of producing more complete and consistent indexes. Early work involves using occurrence characteristics of contents words [Borko, 1970]. Later people came to realize that indexes are often multi-word terms and their generation might involve more elaborated syntactic analysis on phrasal or sentential level [Salton, 1988; Dillon and McDonald, 1983]. However, a full syntactical approach [Salton, 1988] to this task has real problem with efficiency and coverage for unrestricted text. No viable automatic solution is currently in use. Indexing Chinese books involves another severe obstacle, namely the word segmentation problem. Chinese text consists of a sequence of characters which roughly * This research was supported by ROC National Science Council under Contract NSC 81-0408-E-007-529. correspond to letters in English. However, there are no spaces to mark the beginning and end of a word as in English. Until recently, this problem has been considered difficult to solve without elaborated syntactical and semantic analyses [Chen, 1988]. Recent research advances may lead to the development of viable book indexing methods for Chinese books. These include the availability of efficient and high precision word segmentation methods for Chinese text [Chang et al., 1991; Sproat and Shih, 1990; Wang et al., 1990], the availability of statistical analysis of a Chinese corpus [Liu et al., 1975] and large-scale electronic Chinese dictionaries with part- of-speech information [Chang et al., 1988; BDC, 1992], the corpus-based statistical part-of-speech tagger [Church, 1988; DeRose, 1988; Beale, 1988], as well as phrasal and clausal analyzers [Church 1988; Ejerhed 1990] 2. Problem description As being pointed out in [Salton, 1988], back-of-book indexes may consist of more than one word that are derived from a noun phrase. Given the text of a book, an indexing system, must perform some kind of phrasal and statistical analysis in order to produce a list of candidate indexes and their occurrence statistics in order to generate indexes as shown in …

Content truncated to 500 words