A News Story Categorization System

A News Story Categorization System Philip J. Hayes, Laura E. Knecht, and Monica J. Cellio Carnegie Group Inc 650 Commerce Court at Station Square Pittsburgh, PA 15219 Abstract This paper describes a pilot version of a commercial application of natural language processing techniques to the problem of categorizing news stories into broad topic categories. The system does not perform a complete semantic or syntactic analyses of the input stories. Its categorizations are dependent on fragmentary recognition using pattern- matching techniques. The fragments it looks for are determined by a set of knowledge-based rules. The accuracy of the system is only slightly lower than that of human categorizers. 1. Introduction The large economic potential of automatic text processing is leading to an increasing interest in its commercial applications. This paper describes a pilot version of a commercial application of natural language processing techniques to the problem of categorizing news stories into broad topic categories. The conventional way to process natural language texts is to have people read them and perform some action based on what they have read. People, for instance, currently categorize news stories for routing purposes and extract information from banking payment telexes so that transactions can be executed. Unfortunately, using people tends to be: • slow - people read text slowly; • expensive - if the volume of text is high, processing it requires the efforts of many people; • inconsistent - it is very hard to get a group of people to make consistent decisions about text. In many cases, the proper processing of text is central to a company's revenue stream, so that improvements in the processing can provide major leverage and justify major contract system expenditures. Automatic text processing offers the possibility of such improvements in all three areas. A single text processing machine can potentially do the job of several people faster, cheaper, and more consistently. This paper describes an implementation of a system to do text categorization. The texts it operates on are news stories, but similar techniques could be employed on electronic mail messages, telex traffic, technical abstracts, etc.. Once categorization has been accomplished, the results can be used to route the texts involved to interested parties or to facilitate later retrieval of the texts from an archival database. The system described here uses the well- established natural language processing technique of pattern-matching [1, 5]. Since the input to the system is an arbitrary news story on any topic whatsoever, no attempt is made to perform a complete syntactic or semantic analysis. Instead, categorization is based on the presence of particular words and phrases in particular lexical contexts. As the more detailed description in Section 3 will make clear, however, the approach used goes well beyond the keyword approaches used in information retrieval (e.g. [6]). In particular, the words and phrases the system looks for and the context in which it looks for them are specified through a modified version of the powerful pattern matching language used in Carnegie Group's Language Craft …

Content truncated to 500 words