Morphological Analysis for Unsegmented Languages using Recurrent Neural Network Language Model

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2292–2297, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Morphological Analysis for Unsegmented Languages using Recurrent Neural Network Language Model Hajime Morita1,2 Daisuke Kawahara1 Sadao Kurohashi1,2 1 Kyoto University 2 CREST, Japan Science and Technology Agency {hmorita, dk, kuro}@i.kyoto-u.ac.jp Abstract We present a new morphological analy- sis model that considers semantic plausi- bility of word sequences by using a re- current neural network language model (RNNLM). In unsegmented languages, since language models are learned from automatically segmented texts and in- evitably contain errors, it is not apparent that conventional language models con- tribute to morphological analysis. To solve this problem, we do not use language mod- els based on raw word sequences but use a semantically generalized language model, RNNLM, in morphological analysis. In our experiments on two Japanese corpora, our proposed model significantly outper- formed baseline models. This result indi- cates the effectiveness of RNNLM in mor- phological analysis. 1 Introduction In contrast to space-delimited languages like En- glish, word segmentation is the first and most cru- cial step for natural language processing (NLP) in unsegmented languages like Japanese, Chinese, and Thai (Kudo et al., 2004; Kaji and Kitsure- gawa, 2014; Shen et al., 2014; Kruengkrai et al., 2006). Word segmentation is usually performed jointly with related analysis: POS tagging for Chi- nese, and POS tagging and lemmatization (anal- ysis of inflected words) for Japanese. Morpho- logical analysis including word segmentation has been widely and actively studied, and for exam- ple, Japanese word segmentation accuracy is in the high 90s. However, we often observe that strange outputs of downstream NLP applications such as machine translation and question answering come from incorrect word segmentations. For example, the state-of-the-art and popu- lar Japanese morphological analyzers, JUMAN (Kurohashi and Kawahara, 2009) and MeCab (Kudo et al., 2004) both analyze “外国人参政権 (foreigner’s right to vote)” not into the correct seg- mentation of (1a), but into the incorrect and awk- ward segmentation of (1b). (1) a. 外国 / 人 foreigner / 参政 / 権 right to vote b. 外国 foreign / 人参 carrot / 政権 regime JUMAN is a rule-based morphological analyzer, defining word-to-word (including inflection) con- nectivities and their scores. MeCab is a supervised morphological analyzer, learning the probabilities of word/POS/inflection sequence from an anno- tated corpus of tens of thousands of sentences. Both systems, however, cannot realize semanti- cally appropriate analysis, and often produce to- tally strange outputs like the above. This paper proposes a semantically appropriate morphological analysis method for unsegmented languages using a language model. For unseg- mented languages, morphological analysis and language modeling form a chicken-and-egg prob- lem. That is, if high-quality morphological analy- sis is available, we can learn a high-quality lan- guage model from a morphologically analyzed large corpus. On the other hand, if a high-quality language model is available, we can achieve high- quality morphological analysis by looking for a segmented word sequence with a large language model score. However, even …

Content truncated to 500 words