AUTOLEX: An Automatic Lexicon Builder for Minority Languages Using an Open Corpus

AUTOLEX: An Automatic Lexicon Builder for Minority Languages Using an Open Corpus Evan Liz C. Buhay a, Marie Joy P. Evardone a , Hansel B. Nocon a, Davis Muhajereen D. Dimalen a, Rachel Edita O. Roxas b a Information Technology Department, School of Computer Studies MSU Iligan Institute of Technology, Tibanga, Iligan City, Philippines 9200 ebuhay, mevardone, hnocon@yahoo.com, davis.dimalen@gmail.com b College of Computer Studies, De La Salle University-Manila rachel.roxas@delasalle.ph Abstract: The aim of this study is to build natural language resources for languages with limited resources or minority languages. Manually building these resources is tedious and costly. These natural language resources such as a language corpora and lexicon will be used for natural language processing research and system development. Tagalog, a minority language was considered in this study as a test bed. This study exploited the use of the WWW to retrieve documents that are written in a minority language. We employed a frequency-based algorithm to build the lexicon. For our evaluation, we considered 260 Tagalog documents extracted from the web as our corpus. From the corpus, the system automatically selected 1,386 candidate unique words based on the threshold (with value of 10) as the lexical entries. Each lexical entry is validated by a language expert. Our evaluation shows an accuracy of 97.84% and only 2.16% error rate. The error was based on incorrectly spelled words or words that are not Tagalog. Keywords: Automatic lexicon builder. 1 Introduction Philippine natural language (NL) resources such as document corpora and lexicons are badly needed in natural language processing research and system development. Building these NL resources requires time and linguistic knowledge. These resources can be built automatically or manually. The manual process of building NL resources such as a lexicon is tedious and is prone to typographical errors and if copied one by one from an existing lexicon may create copyright issues. Thus, this study implemented an automated system that builds a language specific lexicon by using an open corpus or the World Wide Web. In our study, lexical enries are determined based on the frequency of the words in the corpus. Words with higher frequency were selected and words that are below a certain threshold were not selected. This kind of lexicon can be used in variety of human language technolog systems, such as word database, word processesors, software for read back by speech synthesis in Text-to-Speech systems and dictation by automatic speech recognition systems (Al- shalabi and Kanaan, 2004). Our approach in retrieving documents from the web was based on our previous study on a resource builder on a closed corpus (Dimalen, 2004). The resource builder is ideal since it is a system that automatically builds a minority language corpus. We extended the resource builder to retrieve documents in an open corpus by using google API which has access to the google search engine database. Both the corpus and the lexicon that were automatically built by our system are natural language resources that are significant to Lexicology. Lexicology is the …

Content truncated to 500 words