A Method for Automatic POS Guessing of Chinese Unknown Words

Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008), pages 705–712 Manchester, August 2008 A Method for Automatic POS Guessing of Chinese Unknown Words Likun Qiu Department of Chinese Language and Literature, Peking University / No.5 Yi- heyuan Road, Haidian District, Beijing, China 100871 NEC Laboratories, China qiulk@pku.edu.cn Changjian Hu, Kai Zhao NEC Laboratories, China / 14F, Build- ing.A, Innovation Plaza, No.1 Tsinghua Science Park, Zhongguancun East Road, Haidian District, Beijing, China 100084 {huchangjian, zhaokai} @research.nec.com.cn Abstract This paper proposes a method for auto- matic POS (part-of-speech) guessing of Chinese unknown words. It contains two models. The first model uses a machine- learning method to predict the POS of unknown words based on their internal component features. The credibility of the results of the first model is then measured. For low-credibility words, the second model is used to revise the first model’s results based on the global con- text information of those words. The ex- periments show that the first model achieves 93.40% precision for all words and 86.60% for disyllabic words, which is a significant improvement over the best results reported in previous studies, which were 89% precision for all words and 74% for disyllabic words. Further, the second model improves the results by 0.80% precision for all words and 1.30% for disyllabic words. © 1 Introduction Since written Chinese does not use blank spaces to denote word boundaries, Chinese word seg- mentation becomes an essential task for natural language processing, as in many other Asian lan- guages (Thai, Japanese, Tibetan, etc.). It is diffi- cult to build a complete dictionary comprising all words, for new words are constantly being cre- ated. As such, unknown words may greatly influ- ence the effectiveness of text processing. Studies ©2008. Licensed under the Creative Commons Attribution- Noncommercial-Share Alike 3.0 Unported license (http://creativecommons.org/licenses/by-nc-sa/3.0/). Some rights reserved. on unknown words include detection, POS guessing, sense classification, etc. Current meth- ods for automatic unknown word detection have been relatively successful and widely used in many systems, yet automatic POS guessing for unknown words still remains a challenge for natural language processing research. The task of POS guessing is quite different from traditional POS tagging. Traditional POS tagging involves assigning a single POS tag to a word token, provided that it is known what POS tag this word can take on in principle. This task requires a lexicon that lists possible POS tags for all words. However, unknown words are not in the lexicon, so the task of POS guessing of un- known words involves the guessing of a correct POS for an unknown word from the whole POS set of the current language. Obviously, tradi- tional methods of POS tagging cannot effectively solve the problem of POS guessing of unknown words. In previous work, two types of features have been used for the task of POS guessing of un- known Chinese words. One type is contextual feature, including local contextual features and global contextual features, and the other is inter- nal component feature. …

Content truncated to 500 words