A Dual-Iterative Method for Concept-Word Acquisition from Large-Scale Chinese Corpora

Proceedings of the Australasian Language Technology Workshop 2005, pages 184–190, Sydney, Australia, December 2005. A Dual-Iterative Method for Concept-Word Acquisition from Large-Scale Chinese Corpora Guogang Tian Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences Graduate University of the Chinese Academy of Sciences Beijing, China 100080 naitgg@hotmail.com Cungen Cao Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences Graduate University of the Chinese Academy of Sciences Beijing, China 100080 cgcao@ict.ac.cn Abstract - This paper proposes a dual-iterative method, a hierarchical inner and outer iteration method (HIO), to acquire concept words from a large-scale, un-segmented Chinese corpus. It has two levels of iteration: the EM-CLS algorithm and the Viterbi-C/S algorithm constitute the inner iteration for generating concept words, and the concept word validation constitutes the outer iteration together with the concept word generation. Through multiple iterations, it integrates the concept word generation and validation into a uniform acquisition process. In the process of acquisition, the HIO method can cope with the problem of over-segmentation, over-combination and data sparseness. The experimental result shows that the HIO method is valid for concept word acquisition that can simultaneously increase the precision and recall rate of concept word acquisition. 1. Introduction Concept word acquisition is an important research in knowledge acquisition from text (KAT) (Cao and Sui, 2003), and it is also the foundation of ontology learning (Maedche, 2002). Its main purpose is to acquire plentiful concept words from text corpora. It is very similar to unknown word recognition (Chen and Bai, 1998), (Feng, Chen, et al., 2004) and term extraction (Bourigault and Jacquemin, 1999). However, there are subtle distinctions among these three researches. Generally, concept word can be classified into three types: proper name, compound word and derived word. Except for these three word types, unknown word recognition also identifies numeric-type compounds, and it does not concern known words listed in a dictionary. Term extraction (Bourigault and Jacquemin, 1999) mainly processes domain texts, and often extracts commonly used professional terms from a specific domain text corpus. Fu and Luke (2003) proposed a two-stage Chinese segmentation system. At the first stage, it segmented the input text according to known words on the basis of 2-gram statistical model, and then identified unknown words at the second stage using a hybrid method which consisted of word context, word composition and word juncture model. Yang and Li (2003) proposed a heuristic method that it generated five rules using mutual information and significance estimation to extract unknown word. Peng and Schuurmans (2001) proposed an unsupervised training method to build probability models that accurately segmented Chinese character sequences into words. It used successive EM phases to learn a good probability model over character strings, and then prunes the model with a mutual information selection criterion to obtain a more accurate word lexicon. Lai and Wu (2000, 2002) proposed a likelihood ratio method to extract possible unknown words or phrases defined by PLUs (phrase-like-unit). The final PLU was decided by two …

Content truncated to 500 words