Unknown Word Extraction for Chinese Documents

Unknown Word Extraction for Chinese Documents Keh-Jiann Chen Institute of Information science, Academia Sinica kchen@iis.sinica.edu.tw Wei-Yun Ma Institute of Information science, Academia Sinica ma@iis.sinica.edu.tw Abstract There is no blank to mark word boundaries in Chinese text. As a result, identifying words is difficult, because of segmentation ambiguities and occurrences of unknown words. Conventionally unknown words were extracted by statistical methods because statistical methods are simple and efficient. However the statistical methods without using linguistic knowledge suffer the drawbacks of low precision and low recall, since character strings with statistical significance might be phrases or partial phrases instead of words and low frequency new words are hardly identifiable by statistical methods. In addition to statistical information, we try to use as much information as possible, such as morphology, syntax, semantics, and world knowledge. The identification system fully utilizes the context and content information of unknown words in the steps of detection process, extraction process, and verification process. A practical unknown word extraction system was implemented which online identifies new words, including low frequency new words, with high precision and high recall rates. 1 Introduction One of the most prominent problems in computer processing of Chinese language is identification of the word sequences of input sentences. There is no blank to mark word boundaries in Chinese text. As a result, identifying words is difficult, because of segmentation ambiguities and occurrences of unknown words (i.e. out-of-vocabulary words). Most papers dealing with the problem of word segmentation focus their attention only on the resolution of ambiguous segmentation. The problem of unknown word identification is considered more difficult and needs to be further investigated. According to an inspection on the Sinica corpus (Chen etc., 1996), a 5 million word Chinese corpus with word segmented, it shows that 3.51% of words are not listed in the CKIP lexicon, a Chinese lexicon with more than 80,000 entries. Identifying Chinese unknown words from a document is difficult; since 1. There is no blank to mark word boundaries; 2. Almost all Chinese characters and words are also morphemes; 3. Morphemes are syntactic ambiguous and semantic ambiguous; 4. Words with same morpho-syntactic structure might have different syntactic categories; 5. No simple rules can enumerate all types of unknown words; 6. Online identification from a short text is even harder, since low frequency unknown words are not identifiable by naive statistical methods. It is difficult to identify unknown words in a text since all Chinese characters can either be a morpheme or a word and there are no blank to mark word boundaries. Therefore without (or even with) syntactic or semantic checking, it is difficult to tell whether a character in a particular context is a part of an unknown word or whether it stands alone as a word. Compound words and proper names are two major types of unknown words. It is not possible to list all of the proper names and compounds neither in a lexicon nor enumeration by morphological rules. Conventionally unknown words were extracted by statistical methods …

Content truncated to 500 words