A Lexicon-Constrained Character Model for Chinese Morphological Analysis

R. Dale et al. (Eds.): IJCNLP 2005, LNAI 3651, pp. 542 – 552, 2005. © Springer-Verlag Berlin Heidelberg 2005 A Lexicon-Constrained Character Model for Chinese Morphological Analysis Yao Meng, Hao Yu, and Fumihito Nishino Fujitsu R&D Center Co., Ltd, Room B1003, Eagle Run Plaza, No. 26 Xiaoyun Road, Chaoyang District, Bejing, 100016, P. R. China {Mengyao, Yu, Nishino}@frdc.fujitsu.com Abstract. This paper proposes a lexicon-constrained character model that com- bines both word and character features to solve complicated issues in Chinese morphological analysis. A Chinese character-based model constrained by a lexicon is built to acquire word building rules. Each character in a Chinese sen- tence is assigned a tag by the proposed model. The word segmentation and part- of-speech tagging results are then generated based on the character tags. The proposed method solves such problems as unknown word identification, data sparseness, and estimation bias in an integrated, unified framework. Preliminary experiments indicate that the proposed method outperforms the best SIGHAN word segmentation systems in the open track on 3 out of the 4 test corpora. Ad- ditionally, our method can be conveniently integrated with any other Chinese morphological systems as a post-processing module leading to significant im- provement in performance. 1 Introduction Chinese morphological analysis is a fundamental problem that has been studied ex- tensively [1], [2], [3], [4], [5], [6], [7], [8]. Researchers make use of word or character features to cope with this problem. However, neither of them seems completely satis- factory. In general, a simple word-based approach can achieve about 90% accuracy for segmentation with a medium-size dictionary. However, since no dictionary includes every Chinese word, the unknown word (or Out Of Vocabulary, OOV) problem [9], [10] can severely affect the performance of word-based approaches. Furthermore, word-based models have an estimation bias when faced with segmentation candidates with different numbers of words. For example, in the standard hidden Markov model, the best result, ∏ = − = = n i i i i i T T pwtpttt pTW T 1 1 1 * ) |)(|... ( argmax ) argmax (| , is related to the number of the words in the segmentation candidates. As such, a candidate with fewer words is preferred over those with more words in the selection process. Therefore, most word- based models are likely to fail when a combinational ambiguity1 sequence is separated into multiple words. 1 A typical segmentation ambiguity, it refers to a situation in which the same Chinese sequence may be one word or several words in different contexts. A Lexicon-Constrained Character Model for Chinese Morphological Analysis 543 Compared with Chinese words, Chinese characters are relatively less unambigu- ous. The Chinese character set is very limited. Therefore, unknown characters occur rarely in a sentence. The grammatical advantages of characters have inspired re- searchers to adopt character features in Chinese morphology and parsing [5], [6], [11], [12]. However, it is difficult to incorporate necessary word features, such as the form of a Chinese word and its fixed part-of-speech tags, in most character-based …

Content truncated to 500 words