A Chunking Strategy Towards Unknown Word Detection in Chinese Word Segmentation

R. Dale et al. (Eds.): IJCNLP 2005, LNAI 3651, pp. 530 – 541, 2005. © Springer-Verlag Berlin Heidelberg 2005 A Chunking Strategy Towards Unknown Word Detection in Chinese Word Segmentation Zhou GuoDong Institute for Infocomm Research, 21 Heng Mui Keng Terrace, Singapore 119613 zhougd@i2r.a-star.edu.sg Abstract. This paper proposes a chunking strategy to detect unknown words in Chinese word segmentation. First, a raw sentence is pre-segmented into a sequence of word atoms 1 using a maximum matching algorithm. Then a chunking model is applied to detect unknown words by chunking one or more word atoms together according to the word formation patterns of the word atoms. In this paper, a discriminative Markov model, named Mutual Information Independence Model (MIIM), is adopted in chunking. Besides, a maximum entropy model is applied to integrate various types of contexts and resolve the data sparseness problem in MIIM. Moreover, an error-driven learning approach is proposed to learn useful contexts in the maximum entropy model. In this way, the number of contexts in the maximum entropy model can be significantly reduced without performance decrease. This makes it possible for further improving the performance by considering more various types of contexts. Evaluation on the PK and CTB corpora in the First SIGHAN Chinese word segmentation bakeoff shows that our chunking approach successfully detects about 80% of unknown words on both of the corpora and outperforms the best-reported systems by 8.1% and 7.1% in unknown word detection on them respectively. 1 Introduction Prior to any linguistic analysis of Chinese text, Chinese word segmentation is the necessary first step and one of major bottlenecks in Chinese information processing since a Chinese sentence is written in a continuous string of characters without obvious separators (such as blanks) between the words. During the past two decades, this research has been a hot topic in Chinese information processing [1-10]. There exist two major problems in Chinese word segmentation: ambiguity resolution and unknown word detection. While n-gram modeling and/or word co- occurrence has been successfully applied to deal with the ambiguity problems [3, 5, 10, 12, 13], unknown word detection has become the major bottleneck in Chinese 1 In this paper, word atoms refer to basic building units in words. For example, the word “计算 机 ” (computer) consists of two word atoms: “ 计 算 ”(computing) and “ 机 ”(machine). Generally, word atoms can either occur independently, e.g. “计算”(computing), or only become a part of a word, e.g. “机”(machine) in the word “计算机” (computer). A Chunking Strategy Towards Unknown Word Detection 531 word segmentation. Currently, almost all Chinese word segmentation systems rely on a word dictionary. The problem is that when the words stored in the dictionary are insufficient, the system's performance will be greatly deteriorated by the presence of words that are unknown to the system. Moreover, manual maintenance of a dictionary is very tedious and time consuming. It is therefore important for a Chinese word segmentation system to identify unknown words from the text automatically. In literature, two categories of …

Content truncated to 500 words