Language Independent Morphological Analysis

Language Independent Morphological Analysis Yamashita, Tatsuo and Matsumoto, Yuji Graduate School of Information Science Nara Institute of Science and Technology { tatuo-y, matsu} @is. aist-nara, ac.jp Abstract This paper proposes a framework of language inde- pendent morphological analysis and mainly concen- trate on tokenization, the first process of morpholog- ical analysis. Although tokenization is usually not regarded as a difficult task in most segmented lan- guages such as English, there are a number of prob- lems in achieving precise treatment of lexical entries. We first introduce the concept of morpho-fragments, which are intermediate units between characters and lexical entries. We describe our approach to resolve problems arising in tokenization so as to attain a language independent morphological analyzer. 1 Introduction The first step in natural language processing is to identify words in a sentence. We call this process a morphological analysis. Various languages exist in the world, and strategies for morphological analysis differ by types of language. Conventionally, mor- phological analyzers have been developed in one an- alyzer for each language approach. This is a lan- guage dependent approach. In contrast, We propose a framework of language independent morphologi- cal analysis system. We employ one analyzer for any language approach. This approach enables a rapid implementation of morphological analysis sys- tems for new languages. We define two types of written languages: one is a segmented language, and the other is a non- segmented language. In non-segmented languages such as Chinese and Japanese, since words are not separated by delimiters such as white spaces, tok- enization is a important and difficult task. In seg- mented languages such as English, since words are seemingly separated by white spaces or punctuation marks, tokenization is regarded as a relatively easy task and little attention has been paid to. Therefore, each language dependent morphological analyzer has its own strategy for tokenization. We call a string defined in the dictionary lexeme. From an algorith- mic point of view, tokenization is regarded as the process of converting an input stream of characters into a stream of lexemes. We assume that a morphological analysis consists of three processes: tokenization, dictionary look- up, and disambiguation. Dictionary look-up gets a string and returns a set of lexemes with part-of- speech information. This implicitly contains lemma- tization. Disambiguation selects the most plausible sequence of lexemes by a use of a rule-base model or a hidden Markov model (HMM)(Manning and Schiitze, 1999). Disambiguation i s already language independent, since it does not process strings di- rectly and therefore will not be taken up. On the other hand, tokenization and dictionary look-up are language dependent and shall be explained more in this paper. We consider problems concerning tokenization of segmented languages in Section 2. To resolve these problem, we first apply the method of non- segmented languages processing to segmented lan- guages (Section 3). However, we do not obtain a satisfactory result. Then, we introduce the con- cept of morpho-fragments to generalize the method of non-segmented language processing (Section 4). The proposed framework …

Content truncated to 500 words