Applying Conditional Random Fields to Japanese Morphological Analysis

Applying Conditional Random Fields to Japanese Morphological Analysis Taku Kudo †∗ Kaoru Yamamoto‡ Yuji Matsumoto † †Nara Institute of Science and Technology 8916-5, Takayama-Cho Ikoma, Nara, 630-0192 Japan ‡CREST JST, Tokyo Institute of Technology 4259, Nagatuta Midori-Ku Yokohama, 226-8503 Japan taku-ku@is.naist.jp, kaoru@lr.pi.titech.ac.jp, matsu@is.naist.jp Abstract This paper presents Japanese morphological analy- sis based on conditional random fields (CRFs). Pre- vious work in CRFs assumed that observation se- quence (word) boundaries were fixed. However, word boundaries are not clear in Japanese, and hence a straightforward application of CRFs is not possible. We show how CRFs can be applied to situations where word boundary ambiguity exists. CRFs offer a solution to the long-standing prob- lems in corpus-based or statistical Japanese mor- phological analysis. First, flexible feature designs for hierarchical tagsets become possible. Second, influences of label and length bias are minimized. We experiment CRFs on the standard testbed corpus used for Japanese morphological analysis, and eval- uate our results using the same experimental dataset as the HMMs and MEMMs previously reported in this task. Our results confirm that CRFs not only solve the long-standing problems but also improve the performance over HMMs and MEMMs. 1 Introduction Conditional random fields (CRFs) (Lafferty et al., 2001) applied to sequential labeling problems are conditional models, trained to discriminate the cor- rect sequence from all other candidate sequences without making independence assumption for fea- tures. They are considered to be the state-of-the-art framework to date. Empirical successes with CRFs have been reported recently in part-of-speech tag- ging (Lafferty et al., 2001), shallow parsing (Sha and Pereira, 2003), named entity recognition (Mc- Callum and Li, 2003), Chinese word segmenta- tion (Peng et al., 2004), and Information Extraction (Pinto et al., 2003; Peng and McCallum, 2004). Previous applications with CRFs assumed that observation sequence (e.g. word) boundaries are fixed, and the main focus was to predict label ∗At present, NTT Communication Science Laboratories, 2-4, Hikaridai, Seika-cho, Soraku, Kyoto, 619-0237 Japan taku@cslab.kecl.ntt.co.jp sequence (e.g. part-of-speech). However, word boundaries are not clear in non-segmented lan- guages. One has to identify word segmentation as well as to predict part-of-speech in morphological analysis of non-segmented languages. In this pa- per, we show how CRFs can be applied to situations where word boundary ambiguity exists. CRFs offer a solution to the problems in Japanese morphological analysis with hidden Markov models (HMMs) (e.g., (Asahara and Matsumoto, 2000)) or with maximum entropy Markov models (MEMMs) (e.g., (Uchimoto et al., 2001)). First, as HMMs are generative, it is hard to employ overlapping fea- tures stemmed from hierarchical tagsets and non- independent features of the inputs such as surround- ing words, word suffixes and character types. These features have usually been ignored in HMMs, de- spite their effectiveness in unknown word guessing. Second, as mentioned in the literature, MEMMs could evade neither from label bias (Lafferty et al., 2001) nor from length bias (a bias occurring because of word boundary ambiguity). Easy se- quences with low entropy are likely to be selected during decoding in MEMMs. The consequence …

Content truncated to 500 words