ChineseBERT: Chinese Pretraining Enhanced by Glyph and Pinyin Information

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 2065–2075 August 1–6, 2021. ©2021 Association for Computational Linguistics 2065 ChineseBERT: Chinese Pretraining Enhanced by Glyph and Pinyin Information Zijun Sun♣, Xiaoya Li♣, Xiaofei Sun♣, Yuxian Meng♣ Xiang Ao♠, Qing He♠, Fei Wu♦ and Jiwei Li♦♣ ♣Shannon.AI, ♦Zhejiang University ♠Key Lab of Intelligent Information Processing of Chinese Academy of Sciences {zijun_sun, xiaoya_li, xiaofei_sun, yuxian_meng, jiwei_li}@shannonai.com {aoxiang,heqing}@ict.ac.cn, wufei@zju.edu.cn Abstract Recent pretraining models in Chinese neglect two important aspects specific to the Chinese language: glyph and pinyin, which carry sig- nificant syntax and semantic information for language understanding. In this work, we pro- pose ChineseBERT, which incorporates both the glyph and pinyin information of Chinese characters into language model pretraining. The glyph embedding is obtained based on different fonts of a Chinese character, being able to capture character semantics from the visual features, and the pinyin embedding char- acterizes the pronunciation of Chinese charac- ters, which handles the highly prevalent het- eronym phenomenon in Chinese (the same character has different pronunciations with dif- ferent meanings). Pretrained on large-scale un- labeled Chinese corpus, the proposed Chine- seBERT model yields significant performance boost over baseline models with fewer train- ing steps. The proposed model achieves new SOTA performances on a wide range of Chinese NLP tasks，including machine read- ing comprehension, natural language infer- ence, text classification, sentence pair match- ing, and competitive performances in named entity recognition and word segmentation.1 1 Introduction Large-scale pretrained models have become a fun- damental backbone for various natural language processing tasks such as natural language under- standing (Liu et al., 2019b), text classification (Reimers and Gurevych, 2019; Chai et al., 2020) and question answering (Clark and Gardner, 2017; Lewis et al., 2020). Apart from English NLP tasks, pretrained models have also demonstrated their ef- fectiveness for various Chinese NLP tasks (Sun et al., 2019, 2020; Cui et al., 2019a, 2020). 1The code and pretrained models are publicly available at https://github.com/ShannonAI/ChineseBert. Since pretraining models are originally designed for English, two important aspects specific to the Chinese language are missing in current large-scale pretraining: glyph-based information and pinyin- based information. For the former, a key aspect that makes Chinese distinguishable from languages such as English, German, is that Chinese is a lo- gographic language. The logographic of charac- ters encodes semantic information. For example, “液(liquid)”, “河(river)” and “湖(lake)” all have the radical “氵(water)”, which indicates that they are all related to water in semantics. Intuitively, the rich semantics behind Chinese character glyphs should enhance the expressiveness of Chinese NLP models. This idea has motivated a variety of of work on learning and incorporating Chinese glyph information into neural models (Sun et al., 2014; Shi et al., 2015; Liu et al., 2017; Dai and Cai, 2017; Su and Lee, 2017; Meng et al., 2019), but not yet large-scale pretraining. For the latter, pinyin, the Romanized sequence of a Chinese character representing its pronuncia- tion(s), is crucial in modeling …

Content truncated to 500 words