Chinese and Japanese Word Segmentation Using Word-Level and Character-Level Information

Chinese and Japanese Word Segmentation Using Word-Level and Character-Level Information Tetsuji Nakagawa Corporate Research and Development Center Oki Electric Industry Co., Ltd. 2−5−7 Honmachi, Chuo-ku, Osaka 541-0053, Japan nakagawa378@oki.com Abstract In this paper, we present a hybrid method for Chinese and Japanese word segmentation. Word-level information is useful for analysis of known words, while character-level informa- tion is useful for analysis of unknown words, and the method utilizes both these two types of information in order to effectively handle known and unknown words. Experimental re- sults show that this method achieves high over- all accuracy in Chinese and Japanese word seg- mentation. 1 Introduction Word segmentation in Chinese and Japanese is an important and difficult task. In these lan- guages, words are not separated by explicit delim- iters, and word segmentation must be conducted first in most natural language processing applica- tions. One of the problems which makes word seg- mentation more difficult is existence of unknown (out-of-vocabulary) words. Unknown words are de- fined as words that do not exist in a system’s dictio- nary. The word segmentation system has no knowl- edge about these unknown words, and determining word boundaries for such words is difficult. Accu- racy of word segmentation for unknown words is usually much lower than that for known words. In this paper, we propose a hybrid method for Chinese and Japanese word segmentation, which utilizes both word-level and character-level infor- mation. Word-level information is useful for anal- ysis of known words, and character-level informa- tion is useful for analysis of unknown words. We use these two types of information at the same time to obtain high overall performance. This paper is organized as follows: Section 2 describes previous work on Chinese and Japanese word segmentation on which our method is based. Section 3 introduces the hybrid method which com- bines word-level and character-level processing. Section 4 shows experimental results of Chinese and Japanese word segmentation. Section 5 discusses related work, and Section 6 gives the conclusion. 2 Previous Work on Word Segmentation Our method is based on two existing methods for Chinese or Japanese word segmentation, and we ex- plain them in this section. 2.1 The Markov Model-Based Method Word-based Markov models are used in English part-of-speech (POS) tagging (Charniak et al., 1993; Brants, 2000). This method identifies POS- tags T = t1, . . . , tn, given a sentence as a word se- quence W = w1, . . . , wn, where n is the number of words in the sentence. The method assumes that each word has a state which is the same as the POS of the word and the sequence of states is a Markov chain. A state t transits to another state s with prob- ability P(s|t), and outputs a word w with probabil- ity P(w|t). From such assumptions, the probability that the word sequence W with parts-of-speech T is generated is P(W, T) = n � i=1 P(witi|w0t0 . . . wi−1ti−1), ≃ n � i=1 P(wi|ti)P(ti|ti−1), (1) …

Content truncated to 500 words