Adaptive Sentence Boundary Disambiguation

Adaptive Sentence Boundary Disambiguation David D. Palmer CS Division, 387 Soda Hall #1776 University of California, Berkeley Berkeley, CA 94720-1776 dpalmer@ cs. berkeley, edu Marti A. Hearst Xerox PARC 3333 Coyote Hill Rd Palo Alto, CA 94304 hearst @parc.xeroz. corn Abstract Labeling of sentence boundaries is a nec- essary prerequisite for many natural lan- guage processing tasks, including part-of- speech tagging and sentence alignment. End-of-sentence punctuation marks are ambiguous; to disambiguate them most systems use brittle, special-purpose regular expression grammars and exception rules. As an alternative, we have developed an ef- ficient, trainable algorithm that uses a lex- icon with part-of-speech probabilities and a feed-forward neural network. This work demonstrates the feasibility of using prior probabilities of part-of-speech assignments, as opposed to words or definite part-of- speech assignments, as contextual infor- mation. After training for less than one minute, the method correctly labels over 98.5% of sentence boundaries in a corpus of over 27,000 sentence-boundary marks. We show the method to be efficient and easily adaptable to different text genres, includ- ing single-case texts. 1 Introduction Labeling of sentence boundaries is a necessary prerequisite for many natural language process- ing (NLP) tasks, including part-of-speech tagging (Church, 1988), (Cutting et al., 1991), and sen- tence alignment (Gale and Church, 1993), (Kay and RSscheisen, 1993). End-of-sentence punctuation marks are ambiguous; for example, a period can de- note an abbreviation, the end of a sentence, or both, as shown in the examples below: (1) The group included Dr. J.M. Freeman and T. Boone Pickens Jr. (2) "This issue crosses party lines and crosses philosophical lines!" said Rep. John Rowland (R., Conn.). Riley (1989) determined that in the Tagged Brown corpus (Francis and Kucera, 1982) about 90% of pe- 78 riods occur at the end of sentences, 10% at the end of abbreviations, and about 0.5% as both abbrevi- ations and sentence delimiters. Note from example (2) that exclamation points and question marks are also ambiguous, since they too can appear at loca- tions other than sentence boundaries. Most robust NLP systems, e.g., Cutting et al. (1991), find sentence delimiters by tokenizing the text stream and applying a regular expression gram- mar with some amount of look-ahead, an abbrevia- tion list, and perhaps a list of exception rules. These approaches are usually hand-tailored to the particu- lar text and rely on brittle cues such as capitalization and the number of spaces following a sentence delim- iter. Typically these approaches use only the tokens immediately preceding and following the punctua- tion mark to be disambiguated. However, more con- text can be necessary, such as when an abbreviation appears at the end of a sentence, as seen in (3a-b): (3a) It was due Friday by 5 p.m. Saturday would be too late. (3b) She has an appointment at 5 p.m. Saturday to get her car fired. or when punctuation occurs in a subsentence within quotation marks or parentheses, as seen in Example (2). Some systems have achieved accurate boundary determination by applying very large manual effort. …

Content truncated to 500 words