Internal and External Evidence in the Identification and Semantic Categorization of Proper Names

Internal and External Evidence in the Identification and Semantic Categorization of Proper Names David D. McDonald mcdonald@cs.brandeis.edu Abstract We describe the proper name recognition and classification facility ("PNV') of the SPARSER natural language understanding system. PNF has been used very successfully in the analysis of unrestricted texts in several sublanguages taken from online news sources. It makes its categoriza- tions on the basis of 'external' evidence from the context of the phrases adjacent to the name as well as 'internal' evidence within the sequence of words and characters. A semantic model of each name and its components is maintained and used for subsequent reference. We describe PNF's operations of delimiting, classifying, and semantically recording the structure of a name; we situate PNF with respect to the related parsing mechanisms within Sparser; and finally we work through an extended example that is typical of the sorts of text we have applied PNF to. 1 Introduction: Internal versus External Evidence It has long been appreciated by people working with proper names in unrestricted written texts that any adequate treatment requires the use of a grammar. We know at this point that the correct identification and semantic categorization of names requires an analysis based on the patterning of orthographic and lexical classes of elements that is analogous to what one finds in the other content-rich, syntactic structure-poor phrase types in the 'periphery' of the language such as numbers, dates, citations, etc. The point of this paper will be to argue that this grammar must be context sensitive, and that it should incorporate a rich semantic model of names and their relationships to individuals. The context-sensitivity requirement derives from the fact that the classification of a proper name involves two complementary kinds of evidence, which we will term 'internal' and 'external'. Internal evidence is derived from within the sequence of words that comprise the name. This can be definitive criteria, such as the presence of known 'incorporation terms' ("Ltd.", "G.m.b.H.") that indicate companies; or heuristic criteria such as abbreviations or known first names often indicating people. Name-internal evidence is the only criteria consid- ered in virtually all of the name recognition systems that are reported as part of state of the art information extraction systems (see e.g. Rau 1991, Alshawi 1992, DARPA 1992), most of 32 which depend on large (~20,000 word) gazetteers and lists of known names for their relatively high performance. By contrast, external evidence is the classificatory criteria provided by the context in which a name appears. The basis for this evidence is the obvious observation that names are just ways to refer to individuals of specific types (people, churches, rock groups, etc.), and that these types have characteristic properties and participate in charact~'isfic events. The presence of these properties or events in the immediate context of a proper name can be used to provide confirm- ing or cfiterial evidence for a name's category. External evidence is analyzed in PNF in terms of substitution contexts and operationalized in terms of context-sensitive …

Content truncated to 500 words