Exploring and Adapting Chinese GPT to Pinyin Input Method

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 1899 - 1909 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Exploring and Adapting Chinese GPT to Pinyin Input Method Minghuan Tan1∗and Yong Dai2∗ and Duyu Tang2† and Zhangyin Feng2 Guoping Huang2 and Jing Jiang1 and Jiwei Li3 and Shuming Shi2 1 Singapore Management University. 2 Tencent AI Lab. 3 Zhejiang University. mhtan.2017@phdcs.smu.edu.sg, jingjiang@smu.edu.sg {yongdai,duyutang,aifeng,donkeyhuang,shumingshi}@tencent.com, jiwei_li@zju.edu.cn Abstract While GPT has become the de-facto method for text generation tasks, its application to pinyin input method remains unexplored. In this work, we make the first exploration to leverage Chinese GPT for pinyin input method. We find that a frozen GPT achieves state-of-the-art performance on perfect pinyin. However, the performance drops dramatically when the input includes abbreviated pinyin. A reason is that an abbreviated pinyin can be mapped to many perfect pinyin, which links to even larger number of Chinese characters. We mitigate this issue with two strategies, in- cluding enriching the context with pinyin and optimizing the training process to help distin- guish homophones. To further facilitate the evaluation of pinyin input method, we create a dataset consisting of 270K instances from fifteen domains. Results show that our ap- proach improves the performance on abbrevi- ated pinyin across all domains. Model analysis demonstrates that both strategies contribute to the performance boost. 1 Introduction GPT (Radford et al., 2018, 2019) is a Transformer- based (Vaswani et al., 2017) language model that predicts tokens in an autoregressive manner. With a generic model architecture and the availability of vast web text data, GPT has been successfully developed for English, Chinese (Du, 2019; Zhang et al., 2021b), and many other languages. It shows extraordinary ability to generate fluent sentences and has been successfully applied to a wide range of natural language generation tasks. However, it remains unexplored to what extent GPT handles Chinese pinyin input method1, which is used by ∗ Work done during internship at Tencent AI Lab. * indicates equal contribution. †Corresponding author. 1https://en.wikipedia.org/wiki/Pinyin_ input_method Character Perfect Pinyin Initial Final 我 wo w o 们 men m en Table 1: Examples of initials and finals for Chinese characters “我们 (we)”. hundreds of millions people when they enter Chi- nese characters on computers and cellphones. Pinyin input method allows users to enter Chi- nese characters based on their pronunciations. Given a pinyin2 as the input, pinyin input method returns a list of Chinese characters pronounced with that pinyin. Fundamental elements of pinyin include initials (声母) and finals (韵母). In most cases, a Chinese character is spelled with one initial followed by one final. For example, as shown in Ta- ble 1, the initial and final for the Chinese character “我 (me)” are w and o, respectively. People may enter perfect pinyin (e.g., “wo men” for “我们”), where initials and finals of all Chinese characters are entered. There are about 420 perfect pinyin in common use. Sometimes, especially when multi- ple Chinese characters are entered at once, people may …

Content truncated to 500 words