Non-parametric Bayesian Segmentation of Japanese Noun Phrases

Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 605–615, Edinburgh, Scotland, UK, July 27–31, 2011. c⃝2011 Association for Computational Linguistics Non-parametric Bayesian Segmentation of Japanese Noun Phrases Yugo Murawaki and Sadao Kurohashi Graduate School of Informatics Kyoto University {murawaki, kuro}@i.kyoto-u.ac.jp Abstract A key factor of high quality word segmenta- tion for Japanese is a high-coverage dictio- nary, but it is costly to manually build such a lexical resource. Although external lexical resources for human readers are potentially good knowledge sources, they have not been utilized due to differences in segmentation cri- teria. To supplement a morphological dictio- nary with these resources, we propose a new task of Japanese noun phrase segmentation. We apply non-parametric Bayesian language models to segment each noun phrase in these resources according to the statistical behavior of its supposed constituents in text. For in- ference, we propose a novel block sampling procedure named hybrid type-based sampling, which has the ability to directly escape a lo- cal optimum that is not too distant from the global optimum. Experiments show that the proposed method efficiently corrects the initial segmentation given by a morphological ana- lyzer. 1 Introduction Word segmentation is the first step of natural lan- guage processing for Japanese, Chinese and Thai because they do not delimit words by white-space. Segmentation for Japanese is a successful field of re- search, achieving the F-score of nearly 99% (Kudo et al., 2004). This success rests on a high-coverage dictionary. Unknown words, or words not covered by the dictionary, are often misidentified. Historically, researchers have devoted exten- sive human resources to build and maintain high- coverage dictionaries (Yokoi, 1995). Since the or- thography of Japanese does not specify a standard for segmentation, researchers define their own crite- ria before constructing lexical resources. For this reason, it is difficult to exploit existing external resources, such as dictionaries and encyclopedias for human readers, where entry words are not seg- mented according to the criteria. Among them, encyclopedias are especially important in that they contain a lot of terms that a morphological dictio- nary fails to cover. Most of these terms are noun phrases and consist of more than one word (mor- pheme). For example, an encyclopedia has an en- try “常山城” (tsuneyama-jou, “Tsuneyama Castle”). According to our segmentation criteria, it consists of two words “常山” (tsuneyama) and “城” (jou). However, the morphological analyzer wrongly seg- ments it into “常” (tsune) and “山城” (yamashiro) because “常山” (tsuneyama) is an unknown word. In this paper, we present the first attempt to uti- lize encyclopedias for word segmentation. We seg- ment each entry noun phrase into words. To do this, we examine the main text of the entry, on the as- sumption that if the noun phrase in question con- sists of more than one word, its constituents appear in the main text either freely or as part of other noun phrases. For “常山城” (tsuneyama-jou), its constituent “常山” (tsune) appears by itself and as constituents of other nouns phrases such …

Content truncated to 500 words