Word Identification for Mandarin Chinese Sentences

WORD IDENTIFICATION FOR MANDARIN CIlINESE SENTENCES Abstract Keh-Jiann Chen Shing-lluan Liu Institute of lnfl~rmation Science Academia Sinica Chinese sentences are composed with string of characters without blanks to mark words. However the basic unit for sentence parsing and understanding is word. Therefore the first step of processing Chinese sentences is to identify the words. The difficulties of identifying words include (l) the identification of com- plex words, such as Determinative-Measure, redupli- cations, derived words etc., (2) the identification of proper names,(3) resolving the ambiguous segmenta- tions. In this paper, we propose the possible solutions for the above difficulties. We adopt a matching algo- rithm with 6 different heuristic rules to resolve the am- biguities and achieve an 99.77% of the success rate. The statistical data supports that the maximal match- ing algorithm is the most effective heuristics. 1. Introduction Chinese sentences arc cx)mposed with string of characters without blanks to mark words. However the basic unit for sentence parsing and understanding is word. Therefore the first step of processing Chinese sentences is to identify the words( i.e. segment the character strings of the sentences into word strings). Most of the current Chinese natural language processing systems include a processor for word iden- tification. Also there are many word segmentation techniques been developed. Usually they use a lexicon with a large set of entries to match input sentences [2,10,12,13,14,21]. It is very often that there are many l~)ssible different successful matchings. Therefore the major focus for word identification were on thc resolu- tion of ambiguities. However many other important as- pects, such as what should be done, in what depth and what are considered to be the correct identifications were totally ignored. High identification rates are claimed to be achieved, but none of them were mea- sured under equal bases. There is no agreement in what extend words are considered to be correctly iden- tified. For instance, compounds occur very often in Chi- nese text, but none of the existing systems except ours pay much attention to identify them. Proper name is another type of words which cannot be listed exhaus- tively in the lexicon. Therefore simple matching algo- rithms can not successfully identify either compounds or proper names. In this paper, we like to raise the ptx~blems and the difficulties in identifying words and suggest the possible solutions. 2. Difficulties in the Identification of Words As we mentioned in the prevkms chapter, theba- sic technique to identify the words is by matching algo- rithms. It requires a well prepared lexicon with suffi- cient amount of lexical entries which covers all of the Chinese words. Iqowcver such a large lexicon is never existing nor will be composed, since the set of words is open ended. Not only because the new words will be generated but because there are unlimited number of compounds. Most of the word identification systems deliberately ignore the problem of compounds and leave the problem unsolved until the stage of parsing. We don't agree their view points and …

Content truncated to 500 words