Automatic Learning of Stemming Rules for the Indonesian Language

Automatic Learning of Stemming Rules for the Indonesian Language Lily Suryana Indradjaja School of Computing National University of Singapore 3 Science Drive 2, Singapore 117543 g0303015@nus.edu.sg Stephane Bressan School of Computing National University of Singapore 3 Science Drive 2, Singapore 117543 steph@nus.edu.sg Abstract We present a method for the automatic learning of stemming rules for the Indonesian language. The learning process uses an unlabelled corpus. In the first phase the candidate (word, stem) pairs are automatically extracted from a set of online documents. This phase uses a dictionary but is nevertheless not trivial because of morphing. In the second phase the rules are induced from the thus obtained list of pairs of words with their respective stems. We evaluate the effectiveness of our method for different sizes of the training set with different settings on the thresholds of the support and confidence of each rule. We discuss how these variables affect the quantity and quality of the rules produced. 1� Introduction The Internet and other modem communication infrastructures can be the media for either the predominance of cultural globalization or for the blooming of cultural diversity and culture interchange. The choice of the kind of information society that we are creating is, in part, in the hands of technologists and researchers devising techniques and designing tools for facilitating the maintenance, management of multi-cultural information. One of the central issues in the management of multicultural information or the management of information in a multicultural context is the processing of natural language. There are still more than 6000 living languages at the dawn of the twenty first century. Tools for the processing of speech and text often require large collections of reference data such as dictionaries, morphological rules, grammars, and phonemes. Such collections are expensive to build since they normally require human expert intervention. Our research is concerned with the economical and therefore semi-automatic or automatic acquisition of such linguistic information necessary for the development of other-than-English or multilingual information systems. Our motivation comes from the desire to build computational linguistics and information retrieval tools for the processing and retrieval of documents written in the Indonesian language. Indonesian is the official language of the republic of Indonesia. Although several hundreds regional languages and dialects are used in the Republic, it is spoken and understood by more than two hundred and forty million people. Indonesian is written using roman script and there are more than 1,000,000,000 documents in Indonesian Language on the public World Wide Web. Stemming is the process of finding the stem (root) of a word, by stripping away the affix attached to the word. In many languages words are often obtained by affixing existing words or roots. Stemming is particularly useful in applications in which morphologically-related words are processed or treated in the same way regardless of their forms, for example in a text classifier, information retrieval system, and in dictionary lookup tool. Studies have shown that in application like information retrieval system, inflectional morphology plays a more important role …

Content truncated to 500 words