A Syllable Based Word Recognition Model for Korean Noun Extraction

A Syllable Based Word Recognition Model for Korean Noun Extraction Do-Gil Lee and Hae-Chang Rim Dept. of Computer Science & Engineering Korea University 1, 5-ka, Anam-dong, Seongbuk-ku Seoul 136-701, Korea �dglee, rim �@nlp.korea.ac.kr Heui-Seok Lim Dept. of Information & Communications Chonan University 115 AnSeo-dong CheonAn 330-704, Korea limhs@infocom.chonan.ac.kr Abstract Noun extraction is very important for many NLP applications such as informa- tion retrieval, automatic text classification, and information extraction. Most of the previous Korean noun extraction systems use a morphological analyzer or a Part- of-Speech (POS) tagger. Therefore, they require much of the linguistic knowledge such as morpheme dictionaries and rules (e.g. morphosyntactic rules and morpho- logical rules). This paper proposes a new noun extrac- tion method that uses the syllable based word recognition model. It finds the most probable syllable-tag sequence of the input sentence by using automatically acquired statistical information from the POS tagged corpus and extracts nouns by detecting word boundaries. Furthermore, it does not require any labor for construct- ing and maintaining linguistic knowledge. We have performed various experiments with a wide range of variables influenc- ing the performance. The experimental results show that without morphological analysis or POS tagging, the proposed method achieves comparable performance with the previous methods. 1 Introduction Noun extraction is a process to find every noun in a document (Lee et al., 2001). In Korean, Nouns are used as the most important terms (features) that express the document in NLP applications such as information retrieval, document categorization, text summarization, information extraction, and etc. Korean is a highly agglutinative language and nouns are included in Eojeols. An Eojeol is a sur- face level form consisting of more than one com- bined morpheme. Therefore, morphological anal- ysis or POS tagging is required to extract Korean nouns. The previous Korean noun extraction methods are classified into two categories: morphological analy- sis based method (Kim and Seo, 1999; Lee et al., 1999a; An, 1999) and POS tagging based method (Shim et al., 1999; Kwon et al., 1999). The mor- phological analysis based method tries to generate all possible interpretations for a given Eojeol by implementing a morphological analyzer or a sim- pler method using lexical dictionaries. It may over- generate or extract inaccurate nouns due to lexical ambiguity and shows a low precision rate. Although several studies have been proposed to reduce the over-generated results of the morphological analy- sis by using exclusive information (Lim et al., 1995; Lee et al., 2001), they cannot completely resolve the ambiguity. The POS tagging based method chooses the most probable analysis among the results produced by the morphological analyzer. Due to the resolution of the ambiguities, it can obtain relatively accurate results. But it also suffers from errors not only produced by a POS tagger but also triggered by the preceding mor- phological analyzer. Furthermore, both methods have serious deficien- 철수는(Cheol-Su-neun) 사람들을(sa-lam-deul-eul) 봤다(bwass-da) 철수(Cheol-Su) 는(neun) 사람들(sa-lam-deul) 을(eul) 봤다(bwass-da) 철수(Cheol-Su) 사람(sa-lam) 들(deul) 을(eul) 보(bo) 았(ass) 다(da) eojeol word morpheme proper noun : person name postposition noun : person noun …

Content truncated to 500 words