The Construction of a Dictionary for a Two-layer Chinese Morphological Analyzer

The Construction of a Dictionary for a Two-layer Chinese Morphological Analyzer Chooi-Ling Goh1, Jia Lü1, Yuchang Cheng1, Masayuki Asahara1, and Yuji Matsumoto1 1 Graduate School of Information Science, Nara Institute of Science and Technology, 8916-5 Takayama, Ikoma, Nara 630-0192, Japan {ling-g,jia-l,yuchan-c,masayu-a,matsu}@is.naist.jp Abstract. We built a morphological analyzer, which can be freely used by anyone for research purpose. In order to build a pratical system, a dictionary with reasonable size is necessary. The initial dictionary is built from the Penn Chinese Treebank corpus v4.0 and contains only 33,438 entries. Since the initial dictionary is quite small, unknown word detection methods are applied to a huge raw text in order to extract new words to be added into the system dictionary. We have successfully constructed a dictionary with 120,769 entries. Finally, we propose a two-layer morphological analyzer to cater for two sets of outputs. The first layer produces the minimal segmentation units defined by us, and the second layer transforms the output of the first layer to the original segmentation units defined by Penn Chinese Treebank. Keywords: Morphological analysis, unknown word, dictionary, corpus, machine-learning 1 Introduction To date, a freely usable Chinese morphological analysis system is still not widely available. Furthermore, since there is no single segmentation standard for all tagged corpora provided by different institutions, some systems are available which are developed by the corpora providers, according to their segmentation standard. As far as we know, there is still no system (freely) available for Penn Chinese Treebank1 (hereafter CTB) standard. Since this treebank is widely used by a lot of researches that do parsing, probably it is a good idea to build a practical morphological analyzer for CTB standard. The initial system that we built contains only 33,438 entries in the system dictionary. To build a practical system, this number is too small. Therefore, we tried to enlarge the system dictionary using unknown word extraction methods. We intend to extract a large amount of unknown words from a huge raw text corpus. Based on our methods, we have successfully increased the system dictionary to 120,769 entries. Although this number is good enough for a practical system but we still hope to add more in the future. 2 New Segmentation Unit In Chinese language processing community, no single segmentation standard is agreeable across different instituitions. In SIGHAN bakeoff [1], we could see that different institutions have provided different segmentation standards. Most of the disagreements in the standards come from the segmentation of morphologically derived words [2] and named entities. For example, some would say that “孩子们/NN” (children) as one word and some would prefer to it as two words, “孩子/NN” and “们 /M”. For named entities such as Chinese person names, whether a string of a surname and a given name should be one word or two words, is also under argument. It would be nice if we can build a system that suits everyone’s needs but it sounds almost impossible. Wu [2] tried to define tree structures to 1 http://www.cis.upenn.edu/~chinese/ 332 …

Content truncated to 500 words