Sub-Character Tokenization for Chinese Pretrained Language Models

Sub-Character Tokenization for Chinese Pretrained Language Models Chenglei Si1,2∗, Zhengyan Zhang1∗, Yingfa Chen1∗, Fanchao Qi1, Xiaozhi Wang1, Zhiyuan Liu1†, Yasheng Wang3, Qun Liu3, Maosong Sun1† 1 NLP Group, DCST, IAI, BNRIST, Tsinghua University, Beijing, China {zy-z19,yingfa-c18,qfc17,wangxz20}@mails.tsinghua.edu.cn {liuzy,sms}@tsinghua.edu.cn 2 University of Maryland, College Park, MD, USA clsi@terpmail.umd.edu 3Huawei Noah’s Ark Lab, Hong Kong, China {wangyasheng,qun.liu}@huawei.com Abstract Tokenization is fundamental to pretrained lan- guage models (PLMs). Existing tokenization methods for Chinese PLMs typically treat each character as an indivisible token. How- ever, they ignore the unique feature of the Chinese writing system where additional lin- guistic information exists below the character level, i.e., at the sub-character level. To utilize such information, we propose sub-character (SubChar for short) tokenization. Specifically, we first encode the input text by converting each Chinese character into a short sequence based on its glyph or pronunciation, and then construct the vocabulary based on the encoded text with sub-word segmentation. Experimen- tal results show that SubChar tokenizers have two main advantages over existing tokeniz- ers: 1) They can tokenize inputs into much shorter sequences, thus improving the com- putational efficiency. 2) Pronunciation-based SubChar tokenizers can encode Chinese ho- mophones into the same transliteration se- quences and produce the same tokenization output, hence being robust to homophone ty- pos. At the same time, models trained with SubChar tokenizers perform competitively on downstream tasks. We release our code and models at https://github.com/thunlp /SubCharTokenization to facilitate future work. 1 Introduction Large-scale Transformer-based pretrained lan- guage models (PLMs) (Devlin et al., 2019; Liu et al., 2019; Lan et al., 2020; Clark et al., 2020; He et al., 2021, inter alia), in which tokenization ∗Equal contribution. †Corresponding authors. plays a fundamental role, have achieved great suc- cess in recent years and attracted wide research interest. The most popular type of tokenization adopted by PLMs is sub-word tokenization, such as byte pair encoding (BPE) (Sennrich et al., 2016), Word- Piece (Schuster and Nakajima, 2012), and uni- gram language model segmentation (Kudo, 2018). Recent Chinese PLMs such as CPM (Zhang et al., 2020, 2021b) adopt this kind of sub-word to- kenization. Apart from sub-word tokenization, many other Chinese PLMs adopt a simple charac- ter tokenizer (CharTokenizer for short) that treats every single Chinese character as a token (Sun et al., 2019; Cui et al., 2019a, 2020, inter alia). However, we believe that both of these ex- isting tokenizers are sub-optimal for Chinese. This is based on the observation that Chinese has unique linguistic characteristics: 1) Chinese has an opaque orthography with ir- regular grapheme-phoneme correspondence (Hao and Yang, 2021). This is in contrast to transparent orthographies like Spanish and Finnish where each letter approximately represents one sound. As a result, utilizing pronunciation information in Chi- nese requires explicit pronunciation encoding. 2) Chinese does not have morphological in- flection, unlike morphologically-rich languages like Russian (Coulmas, 1991). This renders sub- word tokenization less useful since the main ad- vantage of sub-word tokenization comes from the fact that it can split common affixes and root words as separate tokens. …

Content truncated to 500 words