Contrastive Approach towards Text Source Classification based on Top-Bag-of-Word Similarity

Contrastive Approach towards Text Source Classification based on Top-Bag-of-Word Similarity Chu-Ren Huang1,2, and Lung-Hao Lee 2 1Hong Kong Polytechnic University 2Institute of Linguistics, Academia Sinica churen.huang@polyu.edu.hk {churen, lunghao}@gate.sinica.edu.tw Abstract. This paper proposes a method to automatically classify texts from different varieties of the same language. We show that similarity measure is a robust tool for studying comparable corpora of language variations. We take LDC’s Chinese Gigaword Corpus composed of three varieties of Chinese from Mainland China, Singapore, and Taiwan, as the comparable corpora. Top-bag-of-word similarity measures reflect distances among the three varieties of the same language. A Top-bag-of-word similarity based contrastive approach was taken to solve the text source classification problem. Our results show that a contrastive approach using similarity to rule out identity of source and to arrive actual source by inference is more robust that directly confirmation of source by similarity. We show that this approach is robust when applied to other texts. Keywords: Top-bag-of-word similarity, Text Source Classification, Contrastive Approach, Comparable Corpus, Chinese Gigaword. 1. Introduction Comparable corpora are corpora which select similar texts in more than one language or language variety1. These texts are typically gathered during the same time period. Comparable corpora are different from parallel corpora are widely used as resources for statistical machine translation, bilingual lexicons. Comparable corpora overcome the scarcity and limitations of parallel corpora, since sources for original, monolingual texts are much more abundant (Barzilay and Elhadad, 2003; Munteanu et al., 2004; Shao and Ng, 2004; Talvensaari et al., 2007). The degree and nature of lexical similarity and contrast among Mandarin Chinese used in different Chinese speaking societies were widely observed but not thoroughly studied due to the lack of comparable corpora. Recently, LDC’s Chinese Gigaword (2003)contains three sets of monolingual corpora selected according to the same set of criteria but in different language varieties from China, Singapore and Taiwan. We will explore it as a comparable corpus for variations of Chinese in this paper. In particular, we propose a measure of top-bag-of-word similarity for comparing the language variants contained in Chinese GigaWord corpus. Texts from the same period of time from Central News Agency (Taiwan), Xinhua News Agency (PRC) and Lianhe Zaobao (Singapore) are extracted and compared in our study. By comparing these three varieties of Mandarin Chinese, we hope to find the language significant lexical contrasts and meaning variations. We also propose a constrative approach towards automatic text source classification based on co-occurence similarity measures with documents from the same time period of Chinese Gigaword. Experimental resultes indicated that our proposed constrastive approach is reliable and robust. The rest of this paper is organized as follows. Section 2 investigates related literature in word similarity measures in comparable corpus and a brief introduction to Chinese Gigaword. Section 3 describe the text source classification based on co-occurence similarity. Section 4 presnts experimental results and further discussion. Finally, Section 5 concludes this study. 1 Definition of comparable corpus according to EAGLES report, accessed at http://www.ilc.cnr.it/EAGLES/corpustyp/node21.html 404 22nd Pacific Asia Conference on Language, …

Content truncated to 500 words