Mining Japanese Compound Words and Their Pronunciations from Web Pages and Tweets

International Joint Conference on Natural Language Processing, pages 849–853, Nagoya, Japan, 14-18 October 2013. Mining Japanese Compound Words and Their Pronunciations from Web Pages and Tweets Xianchao Wu Baidu Inc. wuxianchao@{gmail, baidu}.com Abstract Mining compound words and their pro- nunciations is essential for Japanese in- put method editors (IMEs). We propose to use a chunk-based dependency parser to mine new words, collocations and predicate-argument phrases from large- scale Japanese Web pages and tweets. The pronunciations of the compound words are automatically rewritten by a statistical ma- chine translation (SMT) model. Experi- ments on applying the mined lexicon to a state-of-the-art Japanese IME system1 show that the precision of Kana-Kanji con- version is significantly improved. 1 Introduction New compound words are appearing everyday. Person names, technical terms and organization names are newly created and used in Web pages such as news, blogs, question-answering systems. Abbreviations, food names and event names are formed and shared in Twitter and Facebook. Min- ing of these new compound words, together with their pronunciations, is an important step for nu- merous natural language processing (NLP) appli- cations. Taking Japanese as an example, the lex- icons containing compound words (in a mixture of Kanjis and Kanas) and their pronunciations (in a sequence of Kanas) significantly influence the accuracies of speech generation (Schroeter et al., 2002) and IME systems (Kudo et al., 2011). In ad- dition, monolingual compound words are shown to be helpful for bilingual SMTs (Liu et al., 2010). In this paper, we mine three types (Figure 1) of new (i.e., not included in given lexicons) Japanese compound words and their pronuncia- tions: (1) words, which are combinations of sin- 1freely downloadable from www.simeji.me for Android and http://ime.baidu.jp/type/ for Windows (a) New word detection (single chunks): (b) Collocations (single chunks): (c) Predicate-argument phrases (double chunks): 時/とき の 流/ながれ を 感じ the passage of time Distressed (when) feeling 心/こころ を 痛/いためる 世界 を ドバイ ショック が 襲っ た the world Dubai (debt) crisis shocked 副/ふく 垢/あか を 鍵/かぎ 垢/あか に 変更 し まし た (my) secondary (twitter) account into non-open account (I) changed Figure 1: Examples of new (compound) words. gle characters and/or shorter words; (2) colloca- tions, which are combinations of words; and (3) predicate-argument phrases, which are combina- tions of chunks constrained by semantic depen- dency relations. The sentences were parsed by a state-of-the-art chunk-based Japanese dependency parser, Cabocha2 (Kudo and Matsumoto, 2002a) which makes use of Mecab3 with IPA dictionary4 for word segmenting, POS tagging, and pronunci- ation annotating. The first sentence in Figure 1 contains two new words which were not correctly recognized by Mecab. We call them “new words”, since new se- mantic meanings are generated by the combina- tion of single characters. There is one Kana col- location in the second sentence. Different from many former researches (Manning and Schütze, 1999; Liu et al., 2009) which only mine colloca- tions of two words, we do not limit the number of words in our collocation lexicon. The third sen- tence contains two …

Content truncated to 500 words