Automatic Acquisition of Basic Katakana Lexicon from a Given Corpus

R. Dale et al. (Eds.): IJCNLP 2005, LNAI 3651, pp. 682 –693, 2005. © Springer-Verlag Berlin Heidelberg 2005 Automatic Acquisition of Basic Katakana Lexicon from a Given Corpus Toshiaki Nakazawa, Daisuke Kawahara, and Sadao Kurohashi University of Tokyo, 7-3-1 Hongo Bunkyo-ku, Tokyo, 113-8656, Japan {nakazawa, kawahara, kuro}@kc.t.u-tokyo.ac.jp Abstract. Katakana, Japanese phonogram mainly used for loan words, is a trou-blemaker in Japanese word segmentation. Since Katakana words are heavily domain-dependent and there are many Katakana neologisms, it is almost impossible to construct and maintain Katakana word dictionary by hand. This paper proposes an automatic segmentation method of Japanese Katakana compounds, which makes it possible to construct precise and concise Katakana word dictionary automati-cally, given only a medium or large size of Japanese corpus of some domain. 1 Introduction Handling words properly is very important for Natural Language Processing. Words are basic unit to assign syntactic/semantic information manually, basic unit to acquire knowledge based on frequencies and co-occurrences, and basic unit to access texts in Information Retrieval. Languages with explicit word boundaries, like white spaces in English, do not suffer from this issue so severely, though it is a bit troublesome to handle compounds and hyphenation appropriately. On the other hand, languages without explicit boundaries such as Japanese always suffer from this issue. Japanese character set and their usage. Here, we briefly explain Japanese character set and their usage. Japanese uses about 6,000 ideogram, Kanji characters, 83 phonogram, Hiragana, and another 86 phonogram, Katakana. Kanji is used for Japanese time-honored nouns (including words imported from China ancient times) and stems of verbs and adjectives; Hiragana is used for function words such as postpositions and auxiliary verbs, and endings of verbs and adjectives; Katakana is used for loan words, mostly from the West, as transliterations. Japanese is very active to naturalize loan words. Neologisms in special/technical domains are often transliterated into Katakana words without translations, or even if there are translations, Katakana transliterations are more commonly used in many cases. For example, ������, transliteration of “computer” is more commonly used than the translation, ���(keisanki). Even for some time-honored Japanese nouns, both Japanese nouns and translitera-tions of their English translations are used together these days, and the use of Automatic Acquisition of Basic Katakana Lexicon from a Given Corpus 683 translit-erations is increasing, such as ������, transliteration of “desk work” vs. �����(tsukue shigoto). Furthermore, some Japanese nouns, typically the names of animals, plants, and food, which can be written in Kanji or Hiragana, are also written in Katakana sometimes [4, 6]. Word segmentation and Katakana words. Let us go back to the word segmentation issue. Japanese word segmentation is performed like this: Japanese words are registered into the dictionary; given an input sentence, all possible words embedded in the sentence and their connections are checked by looking up the dictionary and some connectivity grammar; then the most plausible word sequence is selected. The criteria of selecting the best word sequence were simple heuristic rules preferring longer words in earlier times, and some cost calculation …

Content truncated to 500 words