Self-Organizing n-gram Model for Automatic Word Spacing

Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the ACL, pages 633–640, Sydney, July 2006. c⃝2006 Association for Computational Linguistics Self-Organizing �-gram Model for Automatic Word Spacing Seong-Bae Park Yoon-Shik Tae Se-Young Park Department of Computer Engineering Kyungpook National University Daegu 702-701, Korea �sbpark,ystae,sypark �@sejong.knu.ac.kr Abstract An automatic word spacing is one of the important tasks in Korean language pro- cessing and information retrieval. Since there are a number of confusing cases in word spacing of Korean, there are some mistakes in many texts including news ar- ticles. This paper presents a high-accurate method for automatic word spacing based on self-organizing �-gram model. This method is basically a variant of �-gram model, but achieves high accuracy by au- tomatically adapting context size. In order to find the optimal context size, the proposed method automatically in- creases the context size when the contex- tual distribution after increasing it dose not agree with that of the current context. It also decreases the context size when the distribution of reduced context is sim- ilar to that of the current context. This approach achieves high accuracy by con- sidering higher dimensional data in case of necessity, and the increased compu- tational cost are compensated by the re- duced context size. The experimental re- sults show that the self-organizing struc- ture of �-gram model enhances the basic model. 1 Introduction Even though Korean widely uses Chinese charac- ters, the ideograms, it has a word spacing model unlike Chinese and Japanese. The word spacing of Korean, however, is not a simple task, though the basic rule for it is simple. The basic rule asserts that all content words should be spaced. However, there are a number of exceptions due to various postpositions and endings. For instance, it is diffi- cult to distinguish some postpositions from incom- plete nouns. Such exceptions induce many mis- takes of word spacing even in news articles. The problem of the inaccurate word spacing is that they are fatal in language processing and in- formation retrieval. The incorrect word spacing would result in the incorrect morphological analy- sis. For instance, let us consider a famous Korean sentence: “� �����������.” The true word spacing for this sentence is “���� �#� �#�����.” whose meaning is that my fa- ther entered the room. If the sentence is written as “���#���#�����.”, it means that my father entered the bag, which is totally dif- ferent from the original meaning. That is, since the morphological analysis is the first-step in most NLP applications, the sentences with incorrect word spacing must be corrected for their further processing. In addition, the wrong word spacing would result in the incorrect index for terms in in- formation retrieval. Thus, correcting the sentences with incorrect word spacing is a critical task in Ko- rean information processing. One of the most simple and strong models for automatic word spacing is �-gram model. In spite of the advantages of the �-gram model, its prob- lem should be also considered …

Content truncated to 500 words