A Part of Speech Estimation Method for Japanese Unknown Words using a Statistical Model of Morphology and Context

A Part of Speech Estimation Method for Japanese Unknown Words using a Statistical Model of Morphology and Context Masaaki NAGATA NTT Cyber Space Laboratories 1-1 Hikari-no-oka Yokosuka-Shi Kanagawa, 239-0847 Japan nagata@nttnly, isl. ntt. co. jp Abstract We present a statistical model of Japanese unknown words consisting of a set of length and spelling models classified by the character types that con- stitute a word. The point is quite simple: differ- ent character sets should be treated differently and the changes between character types are very im- portant because Japanese script has both ideograms like Chinese (kanji) and phonograms like English (katakana). Both word segmentation accuracy and part of speech tagging accuracy are improved by the proposed model. The model can achieve 96.6% tag- ging accuracy if unknown words are correctly seg- mented. 1 Introduction In Japanese, around 95% word segmentation ac- curacy is reported by using a word-based lan- guage model and the Viterbi-like dynamic program- ming procedures (Nagata, 1994; Yamamoto, 1996; Takeuchi and Matsumoto, 1997; Haruno and Mat- sumoto, 1997). About the same accuracy is reported in Chinese by statistical methods (Sproat et al., 1996). But there has been relatively little improve- ment in recent years because most of the remaining errors are due to unknown words. There are two approaches to solve this problem: to increase the coverage of the dictionary (Fung and Wu, 1994; Chang et al., 1995; Mori and Nagao, 1996) and to design a better model for unknown words (Nagata, 1996; Sproat et al., 1996). We take the latter approach. To improve word segmenta- tion accuracy, (Nagata, 1996) used a single general purpose unknown word model, while (Sproat et al., 1996) used a set of specific word models such as for plurals, personal names, and transliterated foreign words. The goal of our research is to assign a correct part of speech to unknown word as well as identifying it correctly. In this paper, we present a novel statistical model for Japanese unknown words. It consists of a set of word models for each part of speech and word type. We classified Japanese words into nine orthographic types based on the character types that constitute a word. We find that by making different models for each word type, we can better model the length and spelling of unknown words. In the following sections, we first describe the lan- guage model used for Japanese word segmentation. We then describe a series of unknown word mod- els, from the baseline model to the one we propose. Finally, we prove the effectiveness of the proposed model by experiment. 2 Word Segmentation Model 2.1 Baseline Language Model and Search Algorithm Let the input Japanese character sequence be C = Cl...Cm, and segment it into word sequence W = wl ... wn 1 . The word segmentation task can be de- fined as finding the word segmentation 12d that max- imize the joint probability of word sequence given character sequence P(WIC ). Since the maximiza- tion is carried out with fixed …

Content truncated to 500 words