Extracting Chinese Multi-Word Units from Large-Scale Balanced Corpus

Extracting Chinese Multi-Word Units from Large-Scale Balanced Corpus LIU Jianzhou, HE Tingting Department of Computer Science Central China Normal University 430079, Wuhan, China xxzzsoft@sina.com hett@163.net LIU Xiaohua Center of Modern Educational Technology Huazhong University of Science & Technology 430074, Wuhan, China xx liuxh@sohu.com Abstract Automatic Multi-word Units Extraction is an important issue in Natural Language Processing. This paper has proposed a new statistical method based on a large-scale balanced corpus to extract multi-word units. We have used two improved traditional parameters: mutual information and log-likelihood ratio, and have increased the precision for the top 10,000 words extracted through the method to 80.13%. The results of the research indicate that this method is more efficient and robust than previous multi-word units extraction methods. 1� Introduction Natural language processing is a project based on knowledge, thus human's linguistic knowledge must be stored in the computer and the process of human's comprehending and producing languages be formalized before the computer commands human's linguistic potency. Since multi-word units (words formed with at least two characters) are the primary embodiment of semantics (Pinchuck 1977; Sager 1990), research on these words is the starting point for different natural language processing applications. Automatic multi-word units (MWUs) extraction has great theoretical and practical significance to such language information processing research as information indexing, machine translation, voice recognition, document classification as well as thesaurus compiling. Presently, the rapid developments in different professional fields (e.g. computer science, medicine) mean continuous creation of new MWUs, and it is impossible to list them exhaustively in a lexicon. Therefore, automatic extraction of MWUs is a very important issue. Compared with western languages, as for Chinese there is no space between characters and words are hard to define, thus automatic Chinese MWUs extraction will surely confront even more difficulties. In this paper, we have proposed a statistical method based on a large-scale balance corpus to realize the automatic extraction of MWUs. The goal is to extract sets of words with exact meaning from the corpus. Our method mainly consists of three phases (the first two phases include 3 steps respectively and the third phase includes 4 steps). First, select "seeds" (two character word) ready for extension; then extend these seeds at the front or back by K characters; finally, by comparing these parameters, determine which are MWUs. We have assessed the experiment data by measuring precision rates, and the result indicates that our method is more efficient and robust compared with other approaches. The rest of this paper is structured as follows. In section 2, we describe in detail the method and all statistical parameters used. In section 3, we make a comprehensive analysis and just evaluation of the experimental data. In section 4, we outline the related works as well as their results. Finally, we give out conclusions and introduce part of our later research work in section 5. 2 Multi-word Units Extraction Algorithm Our automatic MWUs extraction algorithm takes three phases. Two improved parameters are applied 282 to measure the association ratio …

Content truncated to 500 words