A Comparative Study of the Effect of Word Segmentation On Chinese Terminology Extraction

A Comparative Study of the Effect of Word Segmentation On Chinese Terminology Extraction Luning Ji, Qin Lu, Wenjie Li, YiRong Chen The Department of Computing, The Hong Kong Polytechnic University, Hong Kong, China Email: {cslji, csluqin, cswjli, csyrchen}@comp.polyu.edu.hk Abstract: Automatic term extraction is the first step towards automatic or semi-automatic update of existing domain knowledge base. Most of the researches applied word segmentation as a preproc- essing step to Chinese term extraction. However, segmentation ambiguity is unavoidable, especially in identifying unknown words for Chinese. In this paper, we discuss the effect and limitations of segmentation to Chinese terminology extraction. Detailed study shows that propagated errors caused by word segmentation have great impact on the result of terminology extraction. Based on our analysis and experiments, it is proven that character-based terminology extraction yields much bet- ter result than that using segmentation as a preprocessing step. Keywords: Word Segmentation, Terminology Extraction, Chinese Corpus 1 Introduction Science and technology has recently generated many new theories, materials, technologies, and concepts within both traditional and novel domains of knowledge. The ever-speedier creation of this knowledge is facilitated and abetted by the wide use of computers and the Internet which allows not only faster speedier and more convenient access to recent knowledge claims but also the continual expansion of the collections of scientific literatures held on the Internet and elsewhere. This domain specific knowledge need to be updated constantly and obviously manual updating which relies on domain experts simply cannot cope with such rapid changes. One potentially useful response to this problem is automatic ter- minology extraction (TE). Automatic term extraction is the first step in the automatic or semi-automatic extraction of terminol- ogy. That is, we must identify terms first before validating that they are domain specific terms, which is the second step called terminology extraction. Most research in the area of term extraction has been car- ried out in European languages, such as English, French and German. Given the well-recognized diffi- culties associated with segmenting Chinese texts that arise from the non-use delimiters to indicate word boundaries, there are two alternative methods for identifying and extracting Chinese terms: one is to use word segmentation as the preprocessing step, referred to as the word-based preprocessing model and the other is to simply identify terms as a string of character patterns no matter what the ultimate number of characters in the term, referred to as the character-based preprocessing model. The choice of which method to use is a matter of some importance as incorrect term identification will inevitably result in incorrect terminology identification. Most of the previous works on Chinese considered word segmentation as the prerequisite step to natural language applications for Chinese. Although previous works in word segmentation have made great progresses, segmentation ambiguity is unavoidable. A key hindrance to segmentation is the the particular difficulty in identifying unknown words for Chinese, which is also directly linked to new term discovery.. This motivated us to look into some language-specific issues: when we build up the …

Content truncated to 500 words