Unknown Word Detection for Chinese by a Corpus-based Learning Method

Computational Linguistics and Chinese Language Processing vol. 3, no. 1, February 1998, pp. 27-44 �Computational Linguistics Society of R. O. C. Unknown Word Detection for Chinese by a Corpus-based Learning Method Keh-Jiann Chen*, Ming-Hong Bai* Abstract One of the most prominent problems in computer processing of the Chinese language is identification of the words in a sentence. Since there are no blanks to mark word boundaries, identifying words is difficult because of segmentation ambiguities and occurrences of out-of-vocabulary words (i.e., unknown words). In this paper, a corpus-based learning method is proposed which derives sets of syntactic rules that are applied to distinguish monosyllabic words from monosyllabic morphemes which may be parts of unknown words or typographical errors. The corpus-based learning approach has the advantages of: 1. automatic rule learning, 2. automatic evaluation of the performance of each rule, and 3. balancing of recall and precision rates through dynamic rule set selection. The experimental results show that the rule set derived using the proposed method outperformed hand-crafted rules produced by human experts in detecting unknown words. 1. Introduction One of the most prominent problems in computer processing of Chinese language is the identification of the words in a sentence. There are no blanks to mark word boundaries in Chinese text. As a result, identifying words is difficult because of segmentation ambi- guities and occurrences of out-of-vocabulary words ( i.e., unknown words). For instance, in (1), the proper name ��� 'Wang, Ying-Xiong' is a typical example of an unknown word, and it has ambiguous segmentation of � 'king' �� 'hero'. Another example in (1) ����� 'university student in Taiwan' also has ambiguous segmentations of �� 'Taiwan' ��� 'university student' , ���� 'National Taiwan University' � 'give birth to' ,and �� 'Taiwan' �� 'university' � 'give birth to' etc.: (1) ��������������� 'Ying-Xiong Wang is a typical university student in Taiwan.' * Institute of Information Sicence, Academia Sinica, Taipei, Taiwan, R. O. C. E-mail: {kchen, evan}@iis.sinica.edu.tw 27 Most of the papers dealing with the problem of word segmentation have focused only on the resolution of ambiguous segmentation. The problem of unknown word identification is considered to be more difficult and needs to be further investigated. According to an inspection of the Sinica corpus [Chen et al., 1996], which is a balanced Chinese corpus with words segmented based on the Chinese word segmentation standard for information processing proposed by ROCLING [Huang et al., 1997], the most productive unknown words are of the following types. 1.1 Types of Unknown Words Unknown words are defined as the words which are not in the lexicon. The following types of unknown words most frequently occur in the Sinica corpus. Table 1 shows the frequency distribution of unknown words of the most frequent 14 categories by exam- ining 3 million-word data from the Sinica corpus. (a) abbreviation (acronym): e.g., �� 'China-fuel' (Nb) and �� 'Taiwan-bus' (Nb). (Please refer to table 1 for the meaning of each category name; for instance, Nb denotes proper names.) It is difficult to identify abbreviations …

Content truncated to 500 words