HMM Parameter Learning for Japanese Morphological Analyzer

HMM Parameter Learning for Japanese Morphological Analyzer Koichi Takeuchi� Yuji Matsumoto Graduate School of Information Science Nara Institute of Science and Technology 8916-5 Takayama, Ikoma, Nara 630-01 Japan {kouit-t, matsu}@is.aist-nara.ac.jp Abstract This paper presents a method to apply Hidden Markov Model (HMM) to parameter learning for Japanese morphological analyzer. We especially emphasize how the following two information sources affect the results of the parameter learning: 1) The initial value of parameters, i.e., the initial probabilities and 2) some grammatical constraints that hold in Japanese sen- tences independently of any domain. First and foremost, a simple application of HMM to Japanese corpus does not give a satisfactory results since word boundaries are not clear in Japanese texts because of lack of word sepa- rators. The first results of the experiments show that initial probabilities learned from correct tagged corpus affects greatly to the results and that a small tagged corpus is enough for the initial probabilities. The second result is that the incorporation of simple grammatical constraints works well in the improvements of the results. The final result gives that the total performance of the HMM-based parameter learning achieves almost the same level as the human developed rule-based Japanese morphological analyzer. 1 Introduction Morphological analysis and part-of-speech tagging is an important preprocessing especially for analyses of unrestricted texts. We have been developing a rule-based Japanese morphological analyzer called JUMAN [8]. The rules are represented as costs to lexical entry and cost to pairs of adjacent parts-of-speech (connectiv- ity cost), which are manually assigned. The cost for a lexical entry reflects the probability of the occurrence of the word, and a connectivity cost of a pair of parts- of-speech reflects the probability of an adjacent occurrence of the pair. Greater cost means less probability. Since those costs vary according to the domain of texts, it requires much ef- fort to estimate them for texts of a new domain. Some statistical methods have been proposed for part-of-speech tagging of English and other Indo-European lan- guages. Church[4] proposed a method to use trigram probabilities obtained from tagged Brown corpus and achieved over 95% precision in English part-of-speech. tagging. Cutting[5] used Hidden Markov Model to estimate probability param- eters for the tagger and achieved 96% precision. This experiment was done on 163 Figure 1: Sample result of Japanese morphological analysis a large scale untagged text. Statistics works well for part-of-speech tagging of a language like English since words are separated by spaces and word order is comparatively more restricted than free word order languages like Japanese and Korean. We have pursued a similar approach based on HMM for Japanese part- of-speech tagging, resulting in a poor performance. The reason is that Japanese sentences do not have word separators, thus, word boundaries are not clear, caus- ing spurious ambiguity in word segmentation. Chang and Chen[l] applied HMM to part-of-speech tagging of Chinese. However, they assumed a word-segmented corpus for the training data. We do not assume a large scale tagged corpora. The reasons are the following: …

Content truncated to 500 words