ZEN: Pre-training Chinese Text Encoder Enhanced by N-gram Representations

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4729–4740 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 4729 ZEN: Pre-training Chinese Text Encoder Enhanced by N-gram Representations Shizhe Diao♦∗, Jiaxin Bai♦∗, Yan Song♠♥†, Tong Zhang♦, Yonggang Wang♣ ♦The Hong Kong University of Science and Technology {sdiaoaa, jbai, tongzhang}@ust.hk ♠The Chinese University of Hong Kong (Shenzhen) ♥Shenzhen Research Institute of Big Data songyan@cuhk.edu.cn ♣Sinovation Ventures {wangyonggang}@chuangxin.com Abstract The pre-training of text encoders normally processes text as a sequence of tokens cor- responding to small text units, such as word pieces in English and characters in Chinese. It omits information carried by larger text gran- ularity, and thus the encoders cannot easily adapt to certain combinations of characters. This leads to a loss of important semantic information, which is especially problematic for Chinese because the language does not have explicit word boundaries. In this paper, we propose ZEN, a BERT-based Chinese (Z) text encoder Enhanced by N-gram represen- tations, where different combinations of char- acters are considered during training, thus po- tential word or phrase boundaries are explic- itly pre-trained and fine-tuned with the charac- ter encoder (BERT). Therefore ZEN incorpo- rates the comprehensive information of both the character sequence and words or phrases it contains. Experimental results illustrated the effectiveness of ZEN on a series of Chi- nese NLP tasks, where state-of-the-art results is achieved on most tasks with requiring less resource than other published encoders. It is also shown that reasonable performance is ob- tained when ZEN is trained on a small corpus, which is important for applying pre-training techniques to scenarios with limited data.1 1 Introduction Pre-trained text encoders (Peters et al., 2018b; De- vlin et al., 2018; Radford et al., 2018, 2019; Yang et al., 2019) have drawn much attention in natural language processing (NLP), because state-of-the- art performance can be obtained for many NLP tasks using such encoders. In general, these en- coders are implemented by training a deep neural *Work done during the internship at Sinovation Ventures. †Corresponding author. 1The code and pre-trained models of ZEN are available at https://github.com/sinovation/ZEN. model on large unlabeled corpora. Although the use of big data brings success to these pre-trained encoders, it is still unclear whether existing en- coders have effectively leveraged all useful infor- mation in the corpus. Normally, the pre-training procedures are designed to learn on tokens corre- sponding to small units of texts (e.g., word pieces for English, characters for Chinese) for efficiency and simplicity. However, some important infor- mation carried by larger text units may be lost for certain languages when we use a standard encoder, such as BERT. For example, in Chinese, text se- mantics are greatly affected by recognizing valid n-grams2. This means a pre-trained encoder can po- tentially be improved by incorporating such bound- ary information of important n-grams. Recently, there are studies adapting BERT for Chinese with word information, yet they are lim- ited in maintaining the original BERT structure, augmented with learning from weakly supervised word information …

Content truncated to 500 words