Online Acquisition of Japanese Unknown Morphemes using Morphological Constraints

Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 429–437, Honolulu, October 2008. c⃝2008 Association for Computational Linguistics Online Acquisition of Japanese Unknown Morphemes using Morphological Constraints Yugo Murawaki Sadao Kurohashi Graduate School of Informatics, Kyoto University Yoshida-honmachi, Sakyo-ku, Kyoto, 606-8501, Japan murawaki@nlp.kuee.kyoto-u.ac.jp kuro@i.kyoto-u.ac.jp Abstract We propose a novel lexicon acquirer that works in concert with the morphological ana- lyzer and has the ability to run in online mode. Every time a sentence is analyzed, it detects unknown morphemes, enumerates candidates and selects the best candidates by comparing multiple examples kept in the storage. When a morpheme is unambiguously selected, the lexicon acquirer updates the dictionary of the analyzer, and it will be used in subsequent analysis. We use the constraints of Japanese morphology and effectively reduce the num- ber of examples required to acquire a mor- pheme. Experiments show that unknown mor- phemes were acquired with high accuracy and improved the quality of morphological analy- sis. 1 Introduction Morphological analysis is the first step for most nat- ural language processing applications. In Japanese morphological analysis, segmentation is processed simultaneously with the assignment of a part of speech (POS) tag to each morpheme. Segmentation is a nontrivial task in Japanese because it does not delimit words by white-space. Japanese morphological analysis has successfully adopted dictionary-based approaches (Kurohashi et al., 1994; Asahara and Matsumoto, 2000; Kudo et al., 2004). In these approaches, a sentence is trans- formed into a lattice of morphemes by searching a pre-defined dictionary, and an optimal path in the lattice is selected. This area of research may be considered almost completed, as previous studies reported the F-score of nearly 99% (Kudo et al., 2004). When applied to web texts, however, more errors are made due to unknown morphemes. In previous studies, exper- iments were performed on newspaper articles, but web texts include slang words, informal spelling al- ternates (Nishimura, 2003) and technical terms. For example, the verb “ググる” (gugu-ru, to google) is erroneously segmented into “ググ” (gugu) and “る” (ru). One solution to this problem is to augment the lexicon of the morphological analyzer by extracting unknown morphemes from texts (Mori and Nagao, 1996). In the previous method, a morpheme extrac- tion module worked independently of the morpho- logical analyzer and ran in off-line (batch) mode. It is inefficient because almost all high-frequency morphemes have already been registered to the pre- defined dictionary. Moreover, it is inconvenient when applied to web texts because the web corpus is huge and diverse compared to newspaper corpora. It is not necessarily easy to build subcorpora before lexicon acquisition. Suppose that we want to ana- lyze whaling-related documents. It is unnecessary and probably harmful to acquire morphemes that are irrelevant to the topic. A whaling-related subcorpus should be extracted from the whole corpus but it is not clear how large it must be. We propose a novel lexicon acquirer that works in concert with the morphological analyzer and has the ability to run in online mode. As …

Content truncated to 500 words