An IBM-PC Environment for Chinese Corpus Analysis

AN IBM-PC ENVIRONMENT FOR CHINESE CORPUS ANALYSIS Robert Wing Pong Luk Department of Chinese, Translation and Linguistics, City Polytechnic of Hong Kong Email: CTRWPL92@CPtlKVX.BITNET ABSTRACT This paper describes a set of computer programs for Chinese corpus analysis. These programs include (1) extraction of different characters, bigrams and words; (2) word segmentation based on bigram, maximal-matching and the combined technique; (3) identification of special terms; (4) Chinese concordancing; (5) compiling collocation statistics and (6) evaluation utilities. These programs run on the IBM- PC and batch programs co-ordinate the use of these programs. L INTRODUCTION Corpus analysis utilities are developed and widely available for English. For example, tbe Oxford Concordance program is available for over 10 kinds of mainframe computer (Hockey and Martin, 1987) and the Longman mini-concordancer (Trible and Joncs, 1990) is available for the sales. Further enhancement of these utilities include compiling collocation statistics (Smadja, 1993) and semi-automatic gloassary construction (Tong, 1993). Current research has focused on bilingual corpora (Gale and Clmrch, 1993) with the alignment of parallel-text becomeing an important technical problem. However, there has been little development of corpus analysis tools for Chinese. Since using Chinese fiJr compnters has only become more generally availablein the last ten years, analysis utilities for Chinese are not widely. Although no integrated environment is available for Chinese corpus analysis, many specific analysis programs have been reported in the literature (Kit et al., 1989; Tong et al., 1993; Chang and Chen, 1993; Zhou et al., 1993). A Chinese concordance program a,~d a clmractcr-list extraction program are freely available from a Singapore (FTP) network site (Guo and Liu, 1992). tlowever, the programs run in the SUN workstations while many users, particularly non-computing experts, interact with an IBM-PC in Chinese, rather than a SUN workstation. Tile rapid adwmce of microcomputers has mitigated many storage and processing speed problcms. As for storage, the hard disk capacity can reach as high as 340M bytes which is adequate in comparison with tile demand for a corpus (8M bytes from the PH corpus) and a dictionary (10M bytes). Using a 486 processor, tile processing speed is acceptable if the user expect data to be analyzed over-night, similar to submilting a batch job to a mainframe computer. For example; the utililies we are developing ranked around 42,000 words in a few minutes and produced about one-hundred lines of keyword-in-context in a few seconds for a 4 million character Chinese corpus. This paper describes our effort to develop corpus analysis programs for Chinese. Tile programs are written in Turbo C++, implemented on an IBM-PC (486) with a 120M byte hard disk. The programs are divided into several types: a. format conversion program (norm.exe, phseg.exe, wform.exe) b. extraction of characters, bigrams and words (exsega.exe, exsegmi.exe,bigram.exe, miana.exe, worddh.exe, wral'~ka.exe, wlranka.cxe) c. word segmentation programs (bisegl.exe, whash.exc, bimaxn.exe) d. concordaucing programs (kwic.exe, kwicw.exe) e. collocation statistics programs (cxtract.exe, cxtractw.exe, cxstat.exe, cxstatw.exe) f. gcncral (evaluation) programs (wcomp.exe, scgperf.exe) To run these analysis utilities, a Chinese computing environment called Eten nmst be set up; otherwise Chinese …

Content truncated to 500 words