Balanced Korean Word Spacing with Structural SVM

Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 875–879, October 25-29, 2014, Doha, Qatar. c⃝2014 Association for Computational Linguistics Balanced Korean Word Spacing with Structural SVM Changki Lee* Edward Choi Hyunki Kim *Kangwon National University, Chuncheon-si, Gangwondo, 200-701, Korea Electronics and Telecommunications Research Institute, Daejeon, 305-350, Korea leeck@kangwon.ac.kr mp2893@gmail.com hkk@etri.re.kr Abstract Most studies on statistical Korean word spac- ing do not utilize the information provided by the input sentence and assume that it was completely concatenated. This makes the word spacer ignore the correct spaced parts of the input sentence and erroneously alter them. To overcome such limit, this paper proposes a structural SVM-based Korean word spacing method that can utilize the space information of the input sentence. The experiment on sen- tences with 10% spacing errors showed that our method achieved 96.81% F-score, while the basic structural SVM method only achieved 92.53% F-score. The more the input sentence was correctly spaced, the more accu- rately our method performed. 1 Introduction Automatic word spacing is a task to decide boundaries between words, which is frequently used for correcting spacing errors of text mes- sages, Tweets, or Internet comments before using them in information retrieval applications (Lee and Kim, 2012). It is also often used in post- processing optical character recognition (OCR) or voice recognition (Lee et al., 2007). Except for some Asian languages such as Chinese, Japa- nese and Thai, most languages have explicit word spacing that improves the readability of the text and helps readers better understand the meaning of it. Korean especially has a tricky word spacing system and users often make mis- takes, which makes automatic word spacing an interesting and essential task. In order to easily acquire the training data, most studies on statistical Korean word spacing assume that well-spaced raw text (e.g. newspaper articles) is perfectly spaced and use it for training (Lee and Kim, 2012; Lee and Kim, 2013; Lee et al., 2007; Shim, 2011). This approach, however, cannot observe incorrect spacing since the as- sumption makes the training data devoid of nega- tive example. Consequently, word spacers cannot use the spacing information given by the user, and erroneously alter the correctly spaced parts of the sentence. To utilize the user-given spacing information, a corpus of input sentences and their correctly spaced version is necessary. Construct- ing such corpus, however, requires much time and resource. In this paper, to resolve such issue, we propose a structural SVM-based Korean word spacing model that can utilize the word spacing infor- mation given by the user. We name the proposed model “Balanced Word Spacing Model (BWSM)”. Our approach trains a basic structural SVM-based Korean word spacing model as in (Lee and Kim, 2013), and tries to obtain the sen- tence which achieves the maximum score for the basic model while minimally altering the input sentence. In the following section, we discuss related studies. In Section 3, the proposed method and its relation to Karush-Kuhn-Tucker (KKT) con- dition are explained. The experiment …

Content truncated to 500 words