An Algorithm Combining Statistics-based and Rules-based for Chunk Identification of Chinese Sentences

An Algorithm Combining Statistics-based and Rules-based for Chunk Identification of Chinese Sentences Wang Rongbo, Chi Zheru School of Computer, Hangzhou Dianzi University, Xiasha Higher Education Park, Hangzhou, Zhejiang, China Department of Electronic and Information Engineering, Hong Kong Polytechnic University wangrbo@163.com Abstract. Natural language processing (NLP) is a very hot research domain. One important branch of it is sentence analysis, including Chinese sentence analysis. However, currently, no mature deep analysis theories and techniques are available. An alternative way is to perform shallow parsing on sentences which is very popular in the domain. The chunk identification is a fundamental task for shallow parsing. The purpose of this paper is to characterize a chunk boundary parsing algorithm, using a statistical method combining adjustment rules, which serves as a supplement to traditional statistics-based parsing methods. The experimental results show that the model works well on the small dataset. It will contribute to the sequent processes like chunk tagging and chunk collocation extraction under other topics etc. 1. Introduction Shallow parsing, also called partial parsing or chunk parsing, has become an interesting alternative to full parsing [Abney, 1991]. The main goal of a shallow parser is to divide a sentence into segments which correspond to certain syntactic units. Shallow parsing including chunking and tagging can be done by dividing a non-restricted sentence into non-overlapping segments in an efficient and reliable way although the detailed information from a full parsing is lost. In recent years, many methods are proposed and applied into shallow parsing of different languages. In reference [Antonio Molina, Ferran Pla, 2002], a specialized HMMs is proposed for English sentences to solve different shallow parsing tasks as a tagging problem. At the same time, many methods are proposed for Chinese chunk parsing [Li Heng, etc., 2004a; Li Sujian, etc., 2002; Zhou Qiang, etc., 1996] and Chinese chunk identification [Li Heng, etc., 2004b; Liu Fang, etc., 2000; Xi Chenhai and Sun Maosong, 2002]. In this paper, we propose a chunk description scheme of Chinese sentences and a method combining statistics-based and rule-based on how to identify all chunks in a Chinese sentence. In our scheme, the essence of chunk definition of chunks is the relative meaning independency which is different from other definitions as a new view point. Relative meaning independency requires a chunk to present a relative independent meaning. For example, sentence “我/r 很/d 不/d 喜欢/v 看/v 电视/n” (I do not like to watch TV very much.) can be segmented into three chunks with relative independent meaning, “我”(I), “很不喜欢看” (do not like to watch at all) and “电视” (TV). The main idea of our algorithm can be described as belows. (1) Perform manually chunks identification of Chinese sentences in the training corpus. 446 (2) Obtain the statistical information which will be used in the proposed statistics-based model. (3) Use the model to perform the chunk pre-identification. (4) Obtain the wrong identified instances based on which the error-driven rules learning is performed and a decision tree of rules is constructed. (5) Obtain the right identified …

Content truncated to 500 words