Generating a Category Set of Words Using a Hierarchical Part-of-Speech System and Tagged Corpus

Generating a Category Set of Words Using a Hierarchical Part-of-Speech System and Tagged Corpus Takeyuki KOJIMA and Yoshiyuki KOTANI Dept. of Computer, Information and Communication Science, Tokyo University of Agric. and Tech. 2-24-16 Nakacho, Koganei, Tokyo, Japan {khojima,kotani}©fairy.ei.tuatac.jp Abstract In this paper, we propose a method of generating a proper categorization of morphemes by giving a hierarchical part-of-speech system and a corpus tagged using this part-of- speech system. Our method use hierarchical information in the part-of-speech system and statistical information in the corpus to generate a category set. The statistical information is based on the context of occurrence of categories. First, we specify the format of given information. Then, we describe an algorithm to generate a proper categorization. Finally, we present the results of our experiments in applying this method. We obtaind a moderately proper categorization and found several candidates for improvement. 1 Introduction In natural language processing, it is important to categorize words or morphemes properly. A proper categorization depends on, among other things, the kind of processing task, the domain of target corpus, and the size of the corpus. When categorization is too general, we can not use characters of individual categories. Because the characters of categories hide each other. On the contrary, when categorization is too specific, we can not use the characters of categories also. Because a low frequency of a category decreases the reliability of characters of the category. Past researches have proposed categorizations and tagsets for differnt purposes: morphologi- cal analysis, syntactic analysis or information extraction, etc. While some of the categorizations are made by hand using linguistic knowledge, others are created from annotated corpora auto- matically or semiautomatically. Several researches have forcused on methods of modifing the existing categorization in order to improve the accuracy of their task with respect to their pur- pose (Brants, 1995). Criteria of categorizations in these researches are the accuracies of their tasks. Other researches have proposed an criteria of categorizations of words based on linguistic quality and not processing quality (Dejean, 2000). We propose here a method to decide a proper categorization of morphemes, giving a hier- archical part-of-speech system and a corpus tagged using this part-of-speech system. In other words, our method forcuses on reducing an existing category set using hierarchical information of part-of-speech system and statistical information of the corpus. We recursively subdivide the categories using topdown approach with subdivision score. The subdivision score, which indicates how significant it is to subivide a category, is based on difference between the context of the category and that of its parent category. We explain structure of given information, which consist of a part-of-speech system and a tagged corpus, and a generating category set in Section 2. Section 3 explains the method of generating a category set. Section 4 shows experiments that are performed in order to test the generating algorithm. In the next section, we disscuss the result of the experiments. 217 2 Given Information and Generated Category Set 2.1 Hierarchical Part-of-Speech System The part-of-speech system we …

Content truncated to 500 words