KySS 1.0: a Framework for Automatic Evaluation of Chinese Input Method Engines

International Joint Conference on Natural Language Processing, pages 1195–1201, Nagoya, Japan, 14-18 October 2013. KySS 1.0: a Framework for Automatic Evaluation of Chinese Input Method Engines∗ Zhongye Jia and Hai Zhao† MOE-Microsoft Key Laboratory for Intelligent Computing and Intelligent Systems, Center for Brain-Like Computing and Machine Intelligence Department of Computer Science and Engineering, Shanghai Jiao Tong University 800 Dongchuan Road, Shanghai 200240, China jia.zhongye@gmail.com, zhaohai@cs.sjtu.edu.cn Abstract Chinese Input Method Engine (IME) plays an important role in Chinese language pro- cessing. However, it has been subjected to lacking a proper evaluation metric for a long time. The natural metric for IME is user experience, which is a rather vague goal for research purpose. We propose a novel approach of quantifying user expe- rience by using keystroke count and then correspondingly develop a framework of IME evaluation, which is fast and accu- rate. With the underlying linguistic back- ground, the proposed evaluation frame- work can properly model the user behavior as Chinese is input through English key- board. It is helpful to point out a way to improve the current Chinese IME perfor- mance. 1 1 Introduction Chinese IME is a software solution that enables user to input Chinese into computer with a reason- able size keyboard, by mapping Chinese characters into English letter combinations. Nowadays the majority of Chinese IMEs are pinyin based. Pinyin is originally designed as the phonetic symbol of a Chinese character, using Latin letters as its syllable notation. For example, the pinyin of the Chinese character “爱”(love) is “ài”.2 There are only less ∗This work was partially supported by the National Nat- ural Science Foundation of China (Grant No.60903119, Grant No.61170114, and Grant No.61272248), and the National Ba- sic Research Program of China (Grant No.2009CB320901 and Grant No.2013CB329401). †Corresponding author 1KySS is the abbreviation for: KeyStroke Score 2Chinese is a language with tone, but for nearly all pinyin IMEs, tone marks are omitted since it is inconvenient to input. than 500 pinyin syllables in standard modern Chi- nese, compared with over 6,000 commonly used Chinese characters, which leads to serious ambigu- ity for pinyin-to-character mapping. Other IMEs using various mapping scheme more or less share this same ambiguity problem, although some of them such as five-stroke IME3, may have lower ambiguity but they are very difficult to learn. So at present pinyin IMEs are the most popular, we focus on them in this paper. Modern IMEs are sentence-based(Chen and Lee, 2000) to reduce the ambiguity, which means that the IME generates a character sequence upon a sequence of English letter inputs, e.g. pinyin syllables. This is a non- typical sequence labeling task, as English alphabet input is the original sequence and Chinese charac- ters are target labels. IMEs usually utilize beam search incorporated with language model initial- ized by (Chen and Lee, 2000), then later extended by (Liu and Wang, 2002), (Lee, 2003), and (Zhang, 2007). There are also various methods using con- ventional sequence labeling techniques such as: support vector machine (Jiang et al., 2007), max- …

Content truncated to 500 words