Chinese Word Auto-Confirmation Agent

Chinese Word Auto-Confirmation Agent Jia-Lin Tsai, Cheng-Lung Sung and Wen-Lian Hsu Institute of Information Science, Academia Sinica Nankang, Taipei, Taiwan, R.O.C. {tsaijl,clsung,hsu}@iis.sinica.edu.tw Abstract In various Asian languages, including Chinese, there is no space between words in texts. Thus, most Chinese NLP systems must perform word-segmentation (sentence tokenization). However, successful word-segmentation depends on having a suffi- ciently large lexicon. On the average, about 3% of the words in text are not contained in a lexicon. Therefore, unknown word identification becomes a bottleneck for Chi- nese NLP systems. In this paper, we present a Chinese word auto-confirmation (CWAC) agent. CWAC agent uses a hybrid approach that takes advantage of statistical and linguistic approaches. The task of a CWAC agent is to auto-confirm whether an n-gram input (n ≥ 2) is a Chinese word. We design our CWAC agent to satisfy two criteria: (1) a greater than 98% precision rate and a greater than 75% recall rate and (2) do- main-independent performance (F-measure). These criteria assure our CWAC agents can work automatically without human intervention. Furthermore, by combining sev- eral CWAC agents designed based on different principles, we can construct a multi-CWAC agent through a building-block approach. Three experiments are conducted in this study. The results demonstrate that, for n-gram frequency ≥ 4 in large corpus, our CWAC agent can satisfy the two criteria and achieve 97.82% precision, 77.11% recall, and 86.24% domain-independent F-measure. No existing systems can achieve such a high precision and do- main-independent F-measure. The proposed method is our first attempt for constructing a CWAC agent. We will continue develop other CWAC agents and integrating them into a multi-CWAC agent system. Keywords: natural language processing, word segmentation, unknown word, agent 1 1. Introduction For a human being, efficient word-segmentation (in Chinese) and word sense disambiguation (WSD) arise naturally while a sentence is understood. However, these problems are still difficult for the computer. One reason is that it is hard to create un- seen knowledge in the computer from running texts [Dreyfus 1992]. Here, unseen knowledge refers to contextual meaning and unknown lexicon. Generally, the task of unknown lexicon identification is to identify (1) unknown word (2) unknown word sense, (3) unknown part-of-speech (POS) of a word and (4) unknown word pronunciation. Unknown word identification (UWI) is the most essen- tial step in dealing with unknown lexicons. However, UWI is still quite difficult for Chinese NLP. From [Lin et al. 1993, Chang et al. 1997, Lai et al. 2000, Chen et al. 2002 and Sun et al. 2002], the difficulty of Chinese UWI is caused by the following problems: 1. Just as in other Asian languages, Chinese sentences are composed of strings of characters that do not have blank spaces to mark word boundaries. 2. All Chinese characters can either be a morpheme or a word. Take the Chinese character 花 as an example. It can be either a free morpheme or a word. 3. Unknown words, which usually are compound words and proper names, are too numerous to list in …

Content truncated to 500 words