Use of Mutual Information Based Character Clusters in Dictionary-less Morphological Analysis of Japanese

Use of Mutual Information Based Character Clusters in Dictionary-less Morphological Analysis of Japanese Hideki Kashioka, Yasuhiro Kawata, Yumiko Kinjo, Andrew Finch and Ezra W. Black {kashioka, ykawata, kinjo, finch, black}~itl.atr.co.jp ATR Interpreting Telecommunications Reserach Laboratories Abstract For languages whose character set is very large and whose orthography does not require spac- ing between words, such as Japanese, tokenizing and part-of-speech tagging are often the diffi- cult parts of any morphological analysis. For practical systems to tackle this problem, un- controlled heuristics are primarily used. The use of information on character sorts, however, mitigates this difficulty. This paper presents our method of incorporating character cluster- ing based on mutual information into Decision- Tree Dictionary-less morphological analysis. By using natural classes, we have confirmed that our morphological analyzer has been signifi- cantly improved in both tokenizing and tagging Japanese text. 1 Introduction Recent papers have reported cases of successful part-of-speech tagging with statistical language modeling techniques (Church 1988; Cutting et. al. 1992; Charniak et. al. 1993; Brill 1994; Nagata 1994; Yamamoto 1996). Morphological analysis on .Japanese, however, is more complex because, unlike European languages, no spaces are inserted between words. In fact, even native Japanese speakers place word boundaries incon- sistently. Consequently, individual researchers have been adopting different word boundaries and tag sets based on their own theory-internal justifications. For a practical system to utilize the different word boundaries and tag sets according to the demands of an application, it is necessary to co- ordinate the dictionary used, tag sets, and nu- merous other parameters. Unfortunately, such a task is costly. Furthermore, it is difficult to maintain the accuracy needed to regulate the word boundaries. Also, depending on the pur- pose, new technical terminology may have to be collected, the dictionary has to be coordinated, but the problem of unknown words would still remain. The above problems will arise so long as a dictionary continue to play a principal role. In analyzing Japanese, a Decision-Tree approach with no need for a dictionary (Kashioka, et. al. 1997) has led us to employ, among other param- eters, mutual information (MI) bits of individ- ual characters derived from large hierarchically clustered sets of characters in the corpus. This paper therefore proposes a type of Decision-Tree morphological analysis using the MI of characters but with no need for a dic- tionary. Next the paper describes the use of information on character sorts in morpholog- ical analysis involving the Japanese language, how knowing the sort of each character is use- ful when tokenizing a string of characters into a string of words and when assigning parts-of- speech to them, and our method of clustering characters based on MI bits. Then, it proposes a type of Decision-Tree analysis where the no- tion of MI-based character and word clustering is incorporated. Finally, we move on to an ex- perimental report and discussions. 2 Use of Information on Characters Many languages in the world do not insert a space between words in the written text. Japanese is one of them. …

Content truncated to 500 words