A Maximum Entropy Approach to Identifying Sentence Boundaries

A Maximum Entropy Approach to Identifying Sentence Boundaries Jeffrey C. Reynar and Adwait Ratnaparkhi* Department of Computer and Information Science University of Pennsylvania Philadelphia, Pennsylvania~ USA {jcreynar, adwait } @unagi.cis.upenn.edu Abstract We present a trainable model for identify- ing sentence boundaries in raw text. Given a corpus annotated with sentence bound- aries, our model learns to classify each oc- currence of., ?, and / as either a valid or in- valid sentence boundary. The training pro- cedure requires no hand-crafted rules, lex- ica, part-of-speech tags, or domain-specific information. The model can therefore be trained easily on any genre of English, and should be trainable on any other Roman- alphabet language. Performance is compa- rable to or better than the performance of similar systems, but we emphasize the sim- plicity of retraining for new domains. 1 Introduction The task of identifying sentence boundaries in text has not received as much attention as it deserves. Many freely available natural language processing tools require their input to be divided into sentences, but make no mention of how to accomplish this (e.g. (Brill, 1994; Collins, 1996)). Others perform the division implicitly without discussing performance (e.g. (Cutting et al., 1992)). On first glance, it may appear that using a short list of sentence-final punctuation marks, such as., ?, and /, is sufficient. However, these punctua- tion marks are not used exclusively to mark sen- tence breaks. For example, embedded quotations may contain any of the sentence-ending punctua- tion marks and . is used as a decimal point, in e- mail addresses, to indicate ellipsis and in abbrevia- tions. Both / and ? are somewhat less ambiguous * The authors would like to aclmowledge the support of ARPA grant N66001-94-C-6043, ARO grant DAAH04- 94-G-0426 and NSF grant SBR89-20230. but appear in proper names and may be used mul- tiple times for emphasis to mark a single sentence boundary. Lexically-based rules could be written and excep- tion lists used to disambiguate the difficult cases described above. However, the lists will never be exhaustive, and multiple rules may interact badly since punctuation marks exhibit absorption proper- ties. Sites which logically should be marked with multiple punctuation marks will often only have one ((Nunberg, 1990) as summarized in (White, 1995)). For example, a sentence-ending abbreviation will most likely not be followed by an additional period if the abbreviation already contains one (e.g. note that D. C is followed by only a single . in The presi- dent lives in Washington, D.C.). As a result, we believe that manually writing rules is not a good approach. Instead, we present a solu- tion based on a maximum entropy model which re- quires a few hints about what information to use and a corpus annotated with sentence boundaries. The model trains easily and performs comparably to sys- tems that require vastly more information. Training on 39441 sentences takes 18 minutes on a Sun Ultra Sparc and disambiguating the boundaries in a single Wall Street Journal article requires only 1.4 seconds. 2 Previous …

Content truncated to 500 words