Period Disambiguation with Maxent Model

Period Disambiguation with Maxent Model Chunyu Kit and Xiaoyue Liu Department of Chinese, Translation and Linguistics, City University of Hong Kong, 83 Tat Chee Ave., Kowloon, Hong Kong {ctckit, xyliu0}@cityu.edu.hk Abstract. This paper presents our recent work on period disambigua- tion, the kernel problem in sentence boundary identification, with the maximum entropy (Maxent) model. A number of experiments are con- ducted on PTB-II WSJ corpus for the investigation of how context window, feature space and lexical information such as abbreviated and sentence-initial words affect the learning performance. Such lexical in- formation can be automatically acquired from a training corpus by a learner. Our experimental results show that extending the feature space to integrate these two kinds of lexical information can eliminate 93.52% of the remaining errors from the baseline Maxent model, achieving an F-score of 99.8227%. 1 Introduction Sentence identification is an important issue in practical natural language pro- cessing. It looks simple at first glance since there are a very small number of punctuations, namely, period (“.”), question mark (“?”), and exclamation (“!”), to mark sentence ends in written texts. However, not all of them are consistently used as sentence ends. In particular, the use of the dot “.” is highly ambiguous in English texts. It can be a full stop, a decimal point, or a dot in an abbreviated word, a numbering item, an email address or a ULR. It may be used for other purposes too. Below are a number of examples from PTB-II WSJ Corpus to illustrate its ambiguities. (1) Pierre Vinken, 61 years old, will join the board as a nonexecutive director Nov. 29. (2) The spinoff also will compete with International Business Machines Corp. and Japan’s Big Three -- Hitachi Ltd., NEC Corp. and Fujitsu Ltd. (3) The government’s construction spending figures contrast with a report issued earlier in the week by McGraw-Hill Inc.’s F.W. Dodge Group. Frequently, an abbreviation dot coincides with a full stop, as exemplified by “Ltd.” in (2) above. A number followed by a dot can be a numbering item, or simply a normal number at sentence end. In contrast to “.”, “!” and “?” are rarely ambiguous. They are seldom used for other purposes than exclamation and question marks. Thus, the focus of R. Dale et al. (Eds.): IJCNLP 2005, LNAI 3651, pp. 223–232, 2005. c ⃝ Springer-Verlag Berlin Heidelberg 2005 224 C. Kit and X. Liu sentence identification is on period disambiguation to resolve the ambiguity of “.”: Whenever a dot shows up in a text token, we need to determine whether or not it is a true period. It is a yes-no classification problem that is suitable for various kinds of machine learning technology to tackle. Several approaches were developed for sentence splitting. These approaches can be categorized into three classes: (1) rule-based models consisting of man- ually constructed rules (e.g., in the form of regular expression), supplemented with abbreviation lists, proper names and other relevant lexical resources, as illustrated in [1]; (2) machine learning algorithms, e.g., …

Content truncated to 500 words