Retrieving Collocations From Korean Text

Retrieving Collocations From Korean Text Seonho Kim, Zooil Yang, Mansuk Song {pobi, zooil, mssong}@december.yonsei.ac.kr Dept. of Computer Science, Yonsei University, Seoul, Korea Jung-Ho Ahn jungho@math.yonsei.ac.kr Dept. of Mathematics, Yonsei University, Seoul, Korea Abstract This paper describes a statistical methodology ibr automatically retrieving collocations from POS tagged Korean text using interrupted bi- grams. The free order of Korean makes it hard to identify collocations. We devised four statis- tics, 'frequency', 'randomness', 'condensation', and 'correlation' .to account for the more flexible word order properties of Korean collocations. We extracted meaningful bigrams using an eval- uation ihnction and extended the bigrams to n-gram collocations by generating equivalence sets, a-covers. We view a modeling problem for n-gram collocations as that for clustering of co- hesive words. 1 Introduction There have been many theoretical and applied works related to collocations. A rapidly grow- ing awfilability of copora has attracted interests m statistical methods for automatically extract- mg ¢:o]loeations from textual corpora. However, it is not easy to )dentify the central tendencies of collocation distribution and the borderlines of criteria are often fuzzy because the expres- sions can be of arbitrary lengths in a large va- riety of forms. Getting reliable collocation pat- terns is particularly difficult in Korean which allows arguments to scamble so freely. This paper presents a statistical method using 'in- terrupted bigrams' for automatically retrieving ~:ollocations and idiomatic expressions from Ko- rean text. We suggest several statistics to ac- count for the more flexible word order. If the distribution of a random sample is un- known, we often try to make inferences about its properties described by suitably defined mea- sures. For the properties of arbitrary collocation distribution, four measure statistics: 'high fre- quency', 'condensation', 'randomness', and 'correlation' were devised. Given a morpheme, our system begins by re- trieving the frequency distributions of all bi- grams within window and then meaningful bi- grams are extracted. We produce a-covers to extend them into n-gram collocations 1 According to the definition of Kjellmer and Cowie, a fossilized phrase is a sequence, where the occurrence of one word almost predicts the rest of the phrase and one word predicts a very limited number of words in a semi-fossilized phrase (Kjellmer, 1995) (Cowie, 1981). How- ever, in both fossilized and semi-fossilized types there is a high degree of cohesion among the members of the phrases (Kjellmer, 1995). We consider the cohesions as a-covers that are ob- tained by applying a fuzzy compatibility rela- tion, which satisfies symmetry and reflexivity, to meaningful bigrams. Namely, n-gram collo- cations could be interpreted as equivalent sets of the meaningful bigrams through partitioning. Here, a-covers mean the clustered sets of the meaningful bigrams. 2 Related Works In determining properties of collocations, most of corpus-based approaches accepted that the words of a collocation have a particular statisti- cal distribution(Cruse, 1986). Although previ- ous approaches have shown good results in re- trieving collocations and many properties have been identified, they depend heavily on the fre- quency factor. (Choueka et al., 1983) proposed an …

Content truncated to 500 words