Compound Noun Segmentation Based on Lexical Data Extracted from Corpus

Compound Noun Segmentation Based on Lexical Data Extracted from Corpus* Juntae Yoon jtyoon@linc.cis.upenn.edu IRCS, University of Pennsylvania, 3401 Walnut St., Suite 400A, Philadelphia, PA 19104-6228, USA Abstract Compound noun analysis is one of the crucial prob- lems in Korean language processing because a series of nouns in Korean may appear without white space in real texts, which makes it difficult to identify the morphological constituents. This paper presents an effective method of Korean compound noun segmen- tation based on lexical data extracted from corpus. The segmentation is done by two steps: First, it is based on manually constructed built-in dictionary for segmentation whose data were extracted from 30 million word corpus. Second, a segmentation algo- rithm using statistical data is proposed, where sim- ple nouns and their frequencies are also extracted from corpus. The analysis is executed based on CYK tabular parsing and min-max operation. By exper- iments, its accuracy is about 97.29%, which turns out to be very effective. 1 Introduction Morphological analysis is crucial for processing the agglutinative language like Korean since words in such languages have lots of morphological variants. A sentence is represented by a sequence of eojeols which are the syntactic unit~ delimited by spacing characters in Korean. Unlike in English, an eojeol is not one word but composed of a series of words (content words and functional words). In particu- lar, since an eojeol can often contain more than one noun, we cannot get proper interpretation of the sen- tence or phrase without its accurate segmentation. The problem in compound noun segmentation is that it is not possible to register all compound nouns in the dictionary since nouns are in the open set of words as well as the number of them is very large. Thus, they must be treated as unseen words without a segmentation process. Furthermore, ac- curate compound noun segmentation plays an im- portant role in the application system. Compound noun segmentation is necessarily required for im- proving recall and precision in Korean information * This work was supported by a KOSEF's postdoctoral fel- lowship grant. retrieval, and obtaining better translation in ma- chine translation. For example, suppose that a compound noun 'seol'agsan-gugrib-gongwon(Seol'ag Mountain National Park)' appear in documents. A user might want to retrieve documents about 'seol'agsan(Seol'ag Mountain)', and then it is likely that the documents with seol'agsan-gugrib-gongwon' are also the ones in his interest. Therefore, it should be exactly segmented before indexing in or- der for the documents to be retrieved with the query 'seol'agsan'. Also, to translate 'seol'agsan-gugrib- gongwon' to Seol'ag Mountain National Park, the constituents should be identified first through the process of segmentation. This paper presents two methods for segmentation of compound nouns. First, we extract compound nouns from a large size of corpus, manually divide them into simple nouns and construct the hand built segmentation dictionary with them. The dictionary includes compound nouns which are frequently used and need exceptional process. The number of data are about 100,000. Second, the segmentation algorithm is applied if …

Content truncated to 500 words