Chinese Word Segmentation Based on Contextual Entropy

Chinese Word Segmentation Based on Contextual Entropy Jin Hu Huang School of Informatics and Engineering Flinders University of South Australia GPO Box 2100, Adelaide South Australia 5001 Jin.huang@infoeng. flinders edu.au David Powers School of Informatics and Engineering Flinders University of South Australia GPO Box 2100, Adelaide South Australia 5001 powers@infoeng.flinders.edu.au Abstract Chinese is written without word delimiters so word segmentation is generally considered a key step in processing Chinese texts. This paper presents a new statistical approach to segment Chinese sequences into words based on contextual entropy on both sides of a bigram. It is used to capture the dependency with the left and right contexts in which a bigram occurs. Our approach tries to segment by finding the word boundaries instead of the words. Experimental results show that it is effective for Chinese word segmentation. 1� Introduction Unlike English there is no explicit word boundary in Chinese text. Chinese words can comprise one, two, three or more characters without delimiters. But almost all techniques to Chinese language processing, including machine translation, information retrieval and natural language understanding are based on words. Word segmentation is a key step in Chinese language processing. Several approaches have been developed for Chinese word segmentation. In general two main approaches are widely used: the statistical approach (Gua and Gan, 1994, Sproat and Shih, 1990,1996, , Teaban, Wen, McNab and Witten, 2000, Peng and Schuurmans, 2001) and lexicon-based approach (Yeh and Lee, 1991, Palmer, 1997, Cheng, Yong and Wong, 1999). Some statistical approaches are based on the mutual information (Sproat and Shih, 1990), which only captures the dependency among characters of a word. Some need large pre-tagged corpus for training (Teaban, Wen, McNab and Witten, 2000), which is too expensive to construct at present. Rule-based approaches require a pre-defined word list (dictionary, or lexicon). The coverage of the dictionary is critical for these approaches. Many researches use a combination of approaches (Nie, Jin and Hanna 1994). These are supervised approaches that require extensive human involvement. Some (Sproat and Shih, 1990, de Marcken, 1996, Peng and Schuurmans, 2001) used unsupervised approaches and required little human intervention. It has been long known that contextual information can be used for segmentation (Harris 1955). Dai, Kgoo and Loh (1999) used weighted document frequency as contextual information for Chinese word segmentation. Zhang, Gao and Zhou (2000) used the context dependency for word extraction. Twig and Lee (1994) used contextual entropy to identify unknown Chinese words. Chang, Lin & Su (1995) and Ponte & Croft (1996) used contextual entropy for automatic lexical acquisition. Hutchens & Alder (1998) and Kempe(1999) used the contextual entropy to detect the separator in English and German corpus. In this paper we will present a simple purely statistical approach using contextual entropy for word segmentation. Details about our approach are given in section 1 and 2. 152 2� Contextual Entropy We use a Markov model to estimate the probabilities of symbols of a corpus. The probability of a symbol w with respect to this model M and …

Content truncated to 500 words