Developing an Online Indonesian Corpora Repository

Developing an Online Indonesian Corpora Repository* Ruli Manurung, Bayu Distiawan, and Desmond Darma Putra Faculty of Computer Science, Universitas Indonesia, Depok 16424 maruli@cs.ui.ac.id, bayu.distiawan@ui.ac.id, desmond86@gmail.com Abstract. This paper describes efforts to develop an online repository of Indonesian corpora –and its associated functions and services– that has been designed to support a wide variety of use cases and applications. Two design considerations are ensuring sustainability and accessibility of the corpora, and enabling open enrichment through annotation. The presented model supports OLAC-compliant metadata, is built atop an OAIS-compliant core repository, and exposes data and functionality via RESTful web services. A prototype implementation is presented, which allows users to upload, browse, and search the collection, whose extensible content model currently supports POS tagging. The future plan is for language-independent aspects of the system to be packaged up and released as an open-source package to aid the development of corpora repositories for other languages. Keywords: Indonesian, corpora, annotation, metadata, digital repositories. * This research is part of a collaborative research project funded by ARC Discovery Grant DP0877595. Copyright 2010 by Ruli Manurung, Bayu Distiawan, and Desmond Darma Putra. 1 Introduction Bahasa Indonesia, or just simply Indonesian, is spoken by well over 100 million people, and yet there is a proportionally small amount of available Indonesian language resources that would greatly support linguistics and language technology research. Recent work on Indonesian NLP resources and tools has started to bear results (Adriani and Manurung, 2008), but to further advance research, there is a need for a comprehensive, balanced, and wide-coverage collection of corpora (Arka et al., 2007). Designing and building an online corpora repository is much more complex than simply uploading a set of text files onto a folder accessible over the Internet. For it to be of support to the research community, careful consideration must be paid to the design of standards, protocols, metadata, and architecture. In this paper we present two main desiderata that inform the design of our corpora repository design: the need to ensure sustainability and accessibility of the corpora (Section 2), and the enabling of open enrichment –through annotation– of the primary data (Section 3), before presenting our design and implemented prototype (Section 4). 2 Ensuring sustainability and accessibility Simons and Bird (2008) state that for a language corpus to bring benefit, the design must take into consideration the following properties: • Extant. The digital corpus must store data that must be verifiable to be authentic representations of their corresponding artefacts. To maintain this property of data integrity, issues such as backups, authentication, and reliability of storage infrastructure and communication networks must be considered. • Discoverable. An archive that stores any amount of data will only be useful if its contents can be accessed by parties who need the knowledge to accomplish their goals. Thus, the developed PACLIC 24 Proceedings 243 archive must be easily discoverable and accessible by stakeholders such as government, researchers, industry, or the general public. Simply making it available over a network is not enough to state …

Content truncated to 500 words