Using the Web to Train a Mobile Device Oriented Japanese Input Method Editor

International Joint Conference on Natural Language Processing, pages 1209–1215, Nagoya, Japan, 14-18 October 2013. Using the Web to Train a Mobile Device Oriented Japanese Input Method Editor Xianchao Wu, Rixin Xiao, Xiaoxin Chen Baidu Inc. wuxianchao@gmail.com, {xiaorixin, chenxiaoxin}@baidu.com Abstract This paper describes the construction of a Japanese Input Method Editor (IME) sys- tem for mobile devices, using the large- scale Web pages. We provide the training process of our IME model, n-pos model for local Kana-Kanji conversion and n- gram model for online cloud service. Es- pecially, we propose an online algorithm of mining new compound words, together with the detailed post-filtering process to prune the billion level entries to be used in mobile services. Experiments show that our IME system outperforms two state-of- the-art Japanese IME baselines. We have released our system in a completely free form1 and the system is currently used by millions of users. 1 Introduction Mobile devices such as smart phones and tablet PCs are used by billions of users. For example, Google’s Android system has obtained more than 900 million2 active devices till May 2013. In this paper, we use large-scale Web pages to train a Japanese IME system for mobile devices. Languages such as Chinese and Japanese can not been typed directly using Latin keyboards. This is because there are only 26 English letters in a Latin keyboard, yet the number of Chinese characters are in tens thousands level. Further- more, by connecting several Chinese characters together, the number of yielded words and fre- quently used phrases/idioms are in million level. Thus, language-dependent Input Method Editor (IME) systems are indispensable which maps from combinations of English letters to Chinese char- 1http://simeji.me/ 2http://www.android.com/ acters/words/phrases and Japanese Kanji/Kana se- quences. For example, suppose we want to input a Japanese verb “炒める” (means “fry”, “炒” is a Japanese Kanji character, “める” are two Japanese Kana characters). We first need to know the Kana pronunciation of the Kanji character “炒”, which is “いた”. That is, the verb is pronunced as “いた める”. Then, we need to know the mapping from English letters to Japanese Kanas. Here, “い”, “た”, “め”, “る” respectively correspond to “i”, “ta”, “me”, “ru”, which can be directly typed by using the Latin keyboards. This mapping (e.g., from “i” to “い”) is unique and predefined already. Thus, the real challenge for constructing an IME system for Japanese is to provide the most reason- able Kanji sequence from a given Kana sequence: • one Kanji character can have several correct Kana pronunciations (e.g., “炒” can be pro- nounced as “いた”, “しょう”, “そう”, etc.); • one Kana sequence corresponds to numerous Kanji candidates (such as “いためる” for “炒める”, “痛める” (pain), etc.); It is the context that determines the selection of the most reasonable Kanji sequence. For example, “心をいためる” (“心” = heart, “を” is a Japanese particle right follows an argument and before the argument’s predicate) requires the Kanji to be “痛 める" (heart pain) and “野菜をいためる” needs the Kanji to be “炒める” (fry vegetables). How- ever, …

Content truncated to 500 words