Linguistically Motivated Features for Enhanced Back-of-the-Book Indexing

Proceedings of ACL-08: HLT, pages 932–940, Columbus, Ohio, USA, June 2008. c⃝2008 Association for Computational Linguistics Linguistically Motivated Features for Enhanced Back-of-the-Book Indexing Andras Csomai and Rada Mihalcea Department of Computer Science University of North Texas csomaia@unt.edu,rada@cs.unt.edu Abstract In this paper we present a supervised method for back-of-the-book index construction. We introduce a novel set of features that goes be- yond the typical frequency-based analysis, in- cluding features based on discourse compre- hension, syntactic patterns, and information drawn from an online encyclopedia. In exper- iments carried out on a book collection, the method was found to lead to an improvement of roughly 140% as compared to an existing state-of-the-art supervised method. 1 Introduction Books represent one of the oldest forms of writ- ten communication and have been used since thou- sands of years ago as a means to store and trans- mit information. Despite this fact, given that a large fraction of the electronic documents avail- able online and elsewhere consist of short texts such as Web pages, news articles, scientific reports, and others, the focus of natural language process- ing techniques to date has been on the automa- tion of methods targeting short documents. We are witnessing however a change: more and more books are becoming available in electronic for- mat, in projects such as the Million Books project (http://www.archive.org/details/millionbooks), the Gutenberg project (http://www.gutenberg.org), or Google Book Search (http://books.google.com). Similarly, a large number of the books published in recent years are often available – for purchase or through libraries – in electronic format. This means that the need for language processing tech- niques able to handle very large documents such as books is becoming increasingly important. This paper addresses the problem of automatic back-of-the-book index construction. A back-of- the-book index typically consists of the most impor- tant keywords addressed in a book, with pointers to the relevant pages inside the book. The construc- tion of such indexes is one of the few tasks related to publishing that still requires extensive human la- bor. Although there is a certain degree of computer assistance, consisting of tools that help the profes- sional indexer to organize and edit the index, there are no methods that would allow for a complete or nearly-complete automation. In addition to helping professional indexers in their task, an automatically generated back-of-the- book index can also be useful for the automatic stor- age and retrieval of a document; as a quick reference to the content of a book for potential readers, re- searchers, or students (Schutze, 1998); or as a start- ing point for generating ontologies tailored to the content of the book (Feng et al., 2006). In this paper, we introduce a supervised method for back-of-the-book index construction, using a novel set of linguistically motivated features. The algorithm learns to automatically identify important keywords in a book based on an ensemble of syntac- tic, discourse-based and information-theoretic prop- erties of the candidate concepts. In experiments per- formed on a collection of books and their indexes, the method …

Content truncated to 500 words