Automatic Construction of Japanese KATAKANA Variant List from Large Corpus

Automatic Construction of Japanese KATAKANA Variant List from Large Corpus Takeshi Masuyama Information Technology Center University of Tokyo 7-3-1, Hongo, Bunkyo Tokyo 113-0023 Japan tak@r.dl.itc.u-tokyo.ac.jp Satoshi Sekine Computer Science Department New York University 715 Broadway, 7th floor New York NY 10003 USA sekine@cs.nyu.edu Hiroshi Nakagawa Information Technology Center University of Tokyo 7-3-1, Hongo, Bunkyo Tokyo 113-0023 Japan nakagawa@dl.itc.u-tokyo.ac.jp Abstract This paper presents a method to construct Japanese KATAKANA variant list from large corpus. Our method is useful for information retrieval, information extrac- tion, question answering, and so on, because KATAKANA words tend to be used as “loan words” and the transliteration causes several variations of spelling. Our method consists of three steps. At step 1, our sys- tem collects KATAKANA words from large corpus. At step 2, our system collects can- didate pairs of KATAKANA variants from the collected KATAKANA words using a spelling similarity which is based on the edit distance. At step 3, our system selects vari- ant pairs from the candidate pairs using a semantic similarity which is calculated by a vector space model of a context of each KATAKANA word. We conducted exper- iments using 38 years of Japanese news- paper articles and constructed Japanese KATAKANA variant list with the perfor- mance of 97.4% recall and 89.1% precision. Estimating from this precision, our system can extract 178,569 variant pairs from the corpus. 1 Introduction “Loan words” in Japanese are usually writ- ten by a phonogram type of Japanese charac- ter set, KATAKANA. Because of loan words, the transliteration causes several variations of spelling. Therefore, Japanese KATAKANA words sometimes have several different or- thographies for each original word. For exam- ple, we found at least six different spellings of “spaghetti” in 38 years of Japanese newspaper articles, such as “スパゲッティ,” “スパゲッテ ィー,” “スパゲッテイ,” “スパゲティ,” “スパゲ ティー,” and “スパゲテイ.” The different ex- pression causes problems when we use search engines, question answering systems, and so on (Yamamoto et al., 2003). For example, when we input “スパゲッティ” as a query for a search en- gine or a query for a question answering system, we may not be able to find the web pages or the answers for which we are looking, if a different orthography for “スパゲッティ” is used. We investigated how many documents were retrieved by Google 1 when each Japanese KATAKANA variant of “spaghetti” was used as a query. The result is shown as Table 1. For example, when we inputted “スパゲッテ ィ” as a query of Google, 104,000 documents were retrieved and the percentage was 34.6%, calculated by 104,000 divided by 300,556. From Table 1, we see that each of six variants appears frequently and thus we may not be able to find the web pages for which we are looking. Although we can manually create Japanese KATAKANA variant list, it is a labor-intensive task. In order to solve the problem, we propose an automatic method to construct Japanese KATAKANA variant list from large corpus. Variant # of retrieved documents スパゲッティ 104,000 (34.6%) スパゲッティー 25,400 (8.5%) …

Content truncated to 500 words