Japanese Word Segmentation by Hidden Markov Model

JAPANESE WORD SEGMENTATION BY HIDDEN MARKOV MODEL Constantine P. Papageorgiou BBN Systems and Technologies 70 Fawcett St. Cambridge, MA 02138 ABSTRACT The processing of Japanese text is complicated by the fact that there are no word delimiters. To segment Japanese text, systems typically use knowledge-based methods and large lexicons. This paper presents a novel approach to Japanese word segmentation which avoids the need for Japanese word lexicons and explicit rule bases. The algorithm utilizes a hidden Markov model, a stochastic process, to determine word boundaries. This method has achieved 91% accuracy in segmenting words in a test corpus. 1. INTRODUCTION The segmentation of Japanese words is one of the main challenges in the automatic processing of Japanese text. Unlike English text which has spaces that separate consecutive words, there are no such word boundary indicators in sentences of Japanese (kanji and kana) text. The algorithms used to obtain robust segmentation of Japanese text generally utilize two techniques, lexicon and rule-based approaches. Large lexicons are inevitably used in conjunction with or as a part of the text segmenting algorithms that have been developed. These lexicons are often time consuming to build and are thus not an optimal solution. Knowledge based approaches typically entail a significant amount of human effort in specifying the rules that will be used to determine word segmentation and do not provide sufficient coverage of the language's grammar rules. This paper introduces a hidden Markov model (HMM) which has been developed for Japanese word segmentation. Hidden Markov models are part of the larger class of probabilistic algorithms. These approaches use large sets of data to abstract away the structure of the domain being learned, as probabilities. We will see that with sufficient data such a stochastic process can achieve 91% accuracy in word segmentation which approaches the state-of-the-art in s6gmentation techniques. 2. JAPANESE WORD SEGMENTATION TECHNIQUES Current Japanese word segmentation techniques consistently rely on large lexicons of words in their decision making procedure. A typical word processor will have both a knowledge base which encodes rules of Japanese grammar, as well as a lexicon of over 50,000 words [Mori, et. al. 1990]. Hypothesized segments of incoming text are analyzed to determine if they have any semantics and therefore are more likely to be correct segments; the grammar rules are then invoked to make final segmentation decisions. This technology achieves 95 percent accuracy in segmentation. An alternative approach to Japanese word processing technology is the development of an architecture for Japanese segmentation and part of speech labeling shown in Figure 1 [Matsukawa, et. al. 1993]. Japanese text I JUMAN AMED I POST word segments with part of speech Figure 1: BBN's JUMANIAMEDIPOST word segmentation and part of speech labeling architecture. The architecture of the system is as follows: JUMAN, a rule-based morphological processor which uses a 40,000 word lexicon and a connectivity matrix to determine word segmentation and part of speech labeling, 2. AMED, a rule-based segmentation and part of speech correction system trained on parallel hypothesized and correct …

Content truncated to 500 words