News-Oriented Keyword Indexing with Maximum Entropy Principle

News-Oriented Keyword Indexing with Maximum Entropy Principle. Li Sujian'� Wang Houfeng'� Yu Shiwen'� Xin Chengsheng2 'Institute of Computational Linguistics, Peking University, 100871, Beijing, China Ilisujian, wanghf, yuswl@pku.edu.cn 2 The Information Center of PEOPLE'S DAILY, 100733, Beijing, China csxin@peoplemail.com.cn Abstract In our information era, keywords are very useful to information retrieval, text clustering and so on. News is always a domain attracting a large amount of attention. Aiming at news documents' characteristics and the resources available, this paper proposes to use Maximum Entropy (ME) model to conduct automatic keyword indexing. The focus of ME-based keyword indexing is how to obtain all the candidate items and select useful features for ME model. First, we make use of some relatively mature linguistic techniques and tools to obtain all the possible candidate items. Then, a feature set of ME model will be introduced. At last we test the model, and experimental results are given. 1� Introduction With more and more information flowing into our life, it is very important to lead people to gain more important information in time as short as possible. Keywords are a good solution, which give a brief summary of a document's content. With keywords, people can quickly find what they are most interested in and read them carefully. That will save us a lot of time. In addition, keywords are also useful to the research of information retrieval, text clustering, and topic search (Frank 1999). Manually indexing keywords will cost highly. Thus, automatically indexing keywords from text is of great interests. Several methods have been proposed for extracting English keywords from text. For example, Witten(1999) adopted Naïve Bayes techniques, and Turney (1999) combined decision trees and genetic algorithm in his system. Due to the characteristics of the Chinese language, some researchers adopt the structure of PAT tree and make use of mutual information to obtain keywords (Chien 1997, Yang 2002). Unfortunately, the construction of PAT tree will cost a lot of space and time. In this paper, aiming at the characteristics of news-oriented articles, resources and techniques available, we will introduce ME model to index keywords from text. Section 2 will describe the architecture of the whole system and review the ME model. In section 3, we will introduce how to obtain candidate keywords as input of the ME model. In section 4, we will illustrate the process of how to construct a feature set for the ME model. In section 5, experimental results will be given and analyzed. At last, we will end with the conclusion. 2� System Architecture Keyword indexing can also be called keyword extraction. The definition of a keyword is not restricted to one word in our conception. Here, a keyword might consist of more than one Chinese word, i.e., a term reflecting the main content of a document. In fact one document is composed of a set of terms every of which can be described by many features and must belong to a keyword or not. Thus, the * Supported by the National Grand …

Content truncated to 500 words