CLOWER: A Pre-trained Language Model with Contrastive Learning over Word and Character Representations

Proceedings of the 29th International Conference on Computational Linguistics, pages 3098–3108 October 12–17, 2022. 3098 CLOWER: A Pre-trained Language Model with Contrastive Learning over Word and Character Representations Borun Chen1∗, Hongyin Tang2∗, Jiahao Bu2, Kai Zhang2, Jingang Wang2†, Qifan Wang3, Hai-Tao Zheng1, 4†, Wei Wu2 and Liqian Yu2 1Tsinghua Shenzhen International Graduate School, Tsinghua University 2Meituan 3Meta AI 4Peng Cheng Laboratory cbr20@mails.tsinghua.edu.cn zheng.haitao@sz.tsinghua.edu.cn {tanghongyin,wangjingang02,wuwei30,yuliqian}@meituan.com bujh1994@gmail.com zhangkai.plus@foxmail.com wqfcr@fb.com Abstract Pre-trained Language Models (PLMs) have achieved remarkable performance gains across numerous downstream tasks in natural lan- guage understanding. Various Chinese PLMs have been successively proposed for learning better Chinese language representation. How- ever, most current models use Chinese char- acters as inputs and are not able to encode se- mantic information contained in Chinese words. While recent pre-trained models incorporate both words and characters simultaneously, they usually suffer from deficient semantic interac- tions and fail to capture the semantic relation between words and characters. To address the above issues, we propose a simple yet effec- tive PLM CLOWER, which adopts the Con- trastive Learning Over Word and charactER representations. In particular, CLOWER im- plicitly encodes the coarse-grained information (i.e., words) into the fine-grained representa- tions (i.e., characters) through contrastive learn- ing on multi-grained information. CLOWER is of great value in realistic scenarios since it can be easily incorporated into any existing fine- grained based PLMs without modifying the pro- duction pipelines. Extensive experiments con- ducted on a range of downstream tasks demon- strate the superior performance of CLOWER over several state-of-the-art baselines. 1 Introduction Pre-trained language models (PLMs) have gained tremendous success in the field of natural language processing recently. As a major milestone of PLMs, BERT (Devlin et al., 2019) and its variants (Yang et al., 2019; Liu et al., 2019; Clark et al., 2019) have demonstrated outstanding performance on a wide variety of natural language understanding (NLU) tasks, such as sentiment analysis and ma- chine reading comprehension tasks. The archi- tecture of Transformer (Vaswani et al., 2017) is ∗Equal contribution. †Corresponding authors. typically the foundation for these models, which models the semantic and syntactic relationships be- tween the tokens of the entire input text and learns the contextual representations for each token. Early Chinese PLMs (Sun et al., 2019) often take the sequences of Chinese characters as the input. These models require relatively small vo- cabulary and learn the representations of each char- acter from the corpus, which avoids the Out-Of- Vocabulary problem (Li et al., 2019). However, the meanings of a Chinese word can be totally differ- ent from the meanings of each Chinese character in the word. For example, the meaning of “小 心” (careful) can not be derived from summing the meaning of “小” (small) and “心” (heart). In gen- eral, the phenomenon of semantic gaps between coarse-grained language units and fine-grained lan- guage units (e.g., words & characters, phrases & words) exists not only in Chinese but also in many other languages. To alleviate the gap, prior studies improve the pre-trained models in two directions. One …

Content truncated to 500 words