SINICA CORPUS : Design Methodology for Balanced Corpora

Language, Information and Computation(PACLIC 11),1996, 167-176 SINICA CORPUS: Design Methodology for Balanced Corpora *Keh-Jiann Chen, +Chu-Ren Huang, *Li-Ping Chang, *Hui-Li Hsu *Institute of Information Science, Academia Sinica Institute of History & Philology, Academia Sinica *kchen@iis.sinica.edu.tw , Thschuren@ccvax.sinica.edu.tw Abstract The Academia Sinica Balanced Corpus (Sinica Corpus) is the first balanced Chinese corpus with part-of-speech tagging. The corpus (Sinica 2.0) is open to the research community through the WWW (http://www.sinica.edu.twiftms- binikiwi.sh). Current size of the corpus is 3.5 million words, and the immediate expansion target is five million words. Each text in the corpus is classified and marked according to five criteria: genre, style, mode, topic, and source. The feature values of these classifications are assigned in a hierarchy. Subcorpora can be defined with a specific set of attributes to serve different research purposes. Texts in the corpus are segmented according to the word segmentation standard proposed by the ROC Computational Linguistic Society. Each segmented word is tagged with its part-of-speech. Linguistic patterns and language structures can be extracted from the tagged corpus via a corpus inspection program which has the functions of KWIC searching, filtering, statistics, printing, and collocation. 1. Introduction Corpus-based approaches are fast becoming the most essential and productive technique for theoretical and computational linguistics research (Svartvik 92, Church & Mercer 93). Their impact reaches almost all areas of natural language studies, such as speech processing, information retrieval, lexicography, character recognition etc. Version 2.0 of the Academia Sinica Balanced Corpus (Sinica Corpus) contains 5,345,871 characters, equivalent to 3.5 million words. The Sinica Corpus is the first balanced Chinese corpus with part-of-speech tagging. The following issues have been the major concerns in designing the Sinica Corpus: 1. organization of the corpus, 2. preparation of the corpus, and 3. the use of the corpus. Since a corpus is a sampling of a particular language or sublanguage, which contains an infinite amount of data, it must be representative and balanced if it claims to faithfully represent the facts in that language or sublanguage (Sinclair 87). However there is no reliable criteria for measuring the balance and representation of a corpus. The Brown corpus is the first corpus that claimed to be balanced. It takes the topic domain distribution as the only balancing criterion. In the Sinica Corpus, we explore the possibilities of multi- dimensional attributes and try to balance the corpus in each dimension. The detailed organization of the Sinica Corpus is discussed in section 2. The Sinica Corpus is a word-based Chinese corpus with part-of-speech tagging. Word segmentation, automatic part-of-speech tagging and quality assurance are major concerns after text selection. They are discussed in section 3. The tools for using a tagged corpus are illustrated in section 4. 167 2. Organization of the Sinica Corpus — Texts Selection and Classification We set up a systematic design in order to use and maintain a large amount of texts. The texts are classified according to five attributes: source, mode, style, topic, and genre. Every text is marked with five attribute values. The five …

Content truncated to 500 words