Open Vocabulary Learning for Neural Chinese Pinyin IME

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1584–1594 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 1584 Open Vocabulary Learning for Neural Chinese Pinyin IME Zhuosheng Zhang1,2, Yafang Huang1,2, Hai Zhao1,2,∗ 1Department of Computer Science and Engineering, Shanghai Jiao Tong University 2Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering, Shanghai Jiao Tong University, Shanghai, China {zhangzs, huangyafang}@sjtu.edu.cn, zhaohai@cs.sjtu.edu.cn Abstract Pinyin-to-character (P2C) conversion is the core component of pinyin-based Chinese in- put method engine (IME). However, the con- version is seriously compromised by the am- biguities of Chinese characters corresponding to pinyin as well as the predefined fixed vocab- ularies. To alleviate such inconveniences, we propose a neural P2C conversion model aug- mented by an online updated vocabulary with a sampling mechanism to support open vocab- ulary learning during IME working. Our ex- periments show that the proposed method out- performs commercial IMEs and state-of-the- art traditional models on standard corpus and true inputting history dataset in terms of multi- ple metrics and thus the online updated vocab- ulary indeed helps our IME effectively follows user inputting behavior. 1 Introduction Chinese may use different Chinese characters up to 20,000 so that it is non-trivial to type the Chi- nese character directly from a Latin-style key- board which only has 26 keys (Zhang et al., 2018a). The pinyin as the official romanization representation for Chinese provides a solution that maps Chinese character to a string of Latin al- phabets so that each character has a letter writing form of its own and users can type pinyin in terms of Latin letters to input Chinese characters into a computer. Therefore, converting pinyin to Chinese characters is the most basic module of all pinyin- based IMEs. As each Chinese character may be mapped to a pinyin syllable, it is natural to regard the Pinyin- to-Character (P2C) conversion as a machine trans- ∗ Corresponding author. This paper was partially sup- ported by National Key Research and Development Program of China (No. 2017YFB0304100) and Key Projects of Na- tional Natural Science Foundation of China (U1836222 and 61733011). lation between two different languages, pinyin se- quences and Chinese character sequences (namely Chinese sentence). Actually, such a translation in P2C procedure is even more straightforward and simple by considering that the target Chinese char- acter sequence keeps the same order as the source pinyin sequence, which means that we can decode the target sentence from left to right without any reordering. Meanwhile, there exists a well-known challenge in P2C procedure, too much ambiguity mapping pinyin syllable to character. In fact, there are only about 500 pinyin syllables corresponding to ten thousands of Chinese characters, even though the amount of the commonest characters is more than 6,000 (Jia and Zhao, 2014). As well known, the homophone and the polyphone are quite common in the Chinese language. Thus one pinyin may cor- respond to ten or more Chinese characters on the average. However, pinyin IME …

Content truncated to 500 words