AutoCor: A Query Based Automatic Acquisition of Corpora of Closely-related Languages

AutoCor: A Query Based Automatic Acquisition of Corpora of Closely-related Languages * Davis Muhajereen D. Dimalen a, Rachel Edita O. Roxas b a Information Technology Department, School of Computer Studies Mindanao State University-Iligan Institute of Technology, Tibanga, Iligan City d_dimalen@yahoo.com b College of Computer Studies, De La Salle University-Manila, roxasr@dlsu.edu.ph Abstract. AutoCor is a method for the automatic acquisition and classification of corpora of documents in closely-related languages. It is an extension and enhancement of CorpusBuilder, a system that automatically builds specific minority language corpora from a closed corpus, since some Tagalog documents retrieved by CorpusBuilder are actually documents in other closely-related Philippine languages. AutoCor used the query generation method odds ratio, and introduced the concept of common word pruning to differentiate between documents of closely-related Philippine languages and Tagalog. The performance of the system using with and without pruning are compared, and common word pruning was found to improve the precision of the system. Keywords: document acquisition, document classification. 1. Introduction A corpus is a term used to designate a body of authentic language data that can be used as a basis for linguistic research. 1 It is also applied to a body of language texts that exist in electronic format. It is estimated that there are currently over 4 billion pages on the world wide web (WWW) covering most areas of human endeavor. And as more information are becoming electronically available on the web, we need more effective methods and techniques to access these information. To date, there has been limited effort in taking advantage of this available information on the web for building natural language resources especially for sparse languages (or minority languages) like Tagalog and other Philippine languages. Unfortunately, to manually collect and organize a language specific corpus over the Web is difficult. The process is tedious and time consuming. To add, an expert in linguistics is needed to manually determine the language where the document collected is written. * This project is funded by the Philippine Council for Advanced Science and Technology for Research and Development, Department of Science and Technology, Philippine Government. Copyright 2007 by Davis Muhajereen D. Dimalen, Rachel Edita O. Roxas 1 Orasan, C. and R. Krishnamurthy 2000. An Open Architecture for the Construction and Administration of Corpora. Proceedings of the Second International Conference on Language Resources and Evaluation. pp. 793-800. 146 A system that automatically acquires language specific documents from the Web is one good solution in corpora building. Creating such a system requires knowledge in information retrieval and natural language processing. 2. Automatic Corpora Builder on a Closed and Open Corpus Several components are required for an automatic corpora builder: a set of seed documents, a language modeler, a query generator, a web search engine, and a language filter2. The CorpusBuilder takes advantage of existing search engine database to collect documents from the web3. It iteratively creates new queries to build a corpus in a single minority language. Sets of relevant and non-relevant documents are taken as initial inputs. Relevant …

Content truncated to 500 words