Enhancing Chinese Pre-trained Language Model via Heterogeneous Linguistics Graph

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 1986 - 1996 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Enhancing Chinese Pre-trained Language Model via Heterogeneous Linguistics Graph Yanzeng Li1∗, Jiangxia Cao2,3∗, Xin Cong2,3, Zhenyu Zhang2,3 Bowen Yu2,3, Hongsong Zhu2,3†, Tingwen Liu2,3† 1Wangxuan Institute of Computer Technology, Peking University. Beijing, China 2Institute of Information Engineering, Chinese Academy of Sciences. Beijing, China 3School of Cyber Security, University of Chinese Academy of Sciences. Beijing, China liyanzeng@stu.pku.edu.cn {caojiangxia, congxin, zhangzhenyu1996}@iie.ac.cn {yubowen, zhuhongsong, liutingwen}@iie.ac.cn Abstract Chinese pre-trained language models usually exploit contextual character information to learn representations, while ignoring the lin- guistics knowledge, e.g., word and sentence information. Hence, we propose a task-free en- hancement module termed as Heterogeneous Linguistics Graph (HLG) to enhance Chinese pre-trained language models by integrating linguistics knowledge. Specifically, we con- struct a hierarchical heterogeneous graph to model the characteristics linguistics structure of Chinese language, and conduct a graph- based method to summarize and concretize information on different granularities of Chi- nese linguistics hierarchies. Experimental re- sults demonstrate our model has the ability to improve the performance of vanilla BERT, BERTwwm and ERNIE 1.0 on 6 natural lan- guage processing tasks with 10 benchmark datasets. Further, the detailed experimental analyses have proven that this kind of mod- elization achieves more improvements com- pared with previous strong baseline MWA. Meanwhile, our model introduces far fewer parameters (about half of MWA) and the training/inference speed is about 7x faster than MWA. Our code and processed datasets are available at https://github.com/ lsvih/HLG. 1 Introduction Pre-trained Language Models (PLM) (Peters et al., 2018; Devlin et al., 2019; Radford et al., 2018; Yang et al., 2019) have recently demonstrated the effectiveness on a variety of natural language pro- cessing (NLP) tasks, such as machine translation and text summarization. For a specific downstream task, the parameters of PLMs can be fine-tuned ∗Both authors contributed equally to this work †Corresponding Author with accurately labeled instances or weakly labeled instances of the task to achieve better performance. In recent, there are a series of studies on adapting PLMs for Chinese (Meng et al., 2019; Sun et al., 2019; Cui et al., 2019a; Sun et al., 2020; Wei et al., 2019; Diao et al., 2020; Lai et al., 2021). Many re- searchers introduce the Chinese-specific linguistics knowledge such as word information into PLMs by conducting elaborate self-supervised tasks to pre- train Chinese PLMs from scratch. Nevertheless, pre-training a PLM is computationally expensive and time-consuming since it needs large-scale Chi- nese corpus and heavy computational resources. The high cost makes it difficult for researchers to pre-train a PLM from scratch. An alternative way is to integrate the Chinese- specific linguistics knowledge into pre-trained PLMs in the fine-tuning stage in downstream tasks directly. Following this idea, the task-free enhance- ment module is widely used in the fine-tuning stage by adding an additional adapter in PLMs to in- tegrate external knowledge (Li et al., 2020). As shown in Figure 1, the enhancement module …

Content truncated to 500 words