Dict-BERT: Enhancing Language Model Pre-training with Dictionary

Findings of the Association for Computational Linguistics: ACL 2022, pages 1907 - 1918 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Dict-BERT: Enhancing Language Model Pre-training with Dictionary Wenhao Yu1∗, Chenguang Zhu2, Yuwei Fang2, Donghan Yu3∗, Shuohang Wang2, Yichong Xu2, Michael Zeng2, Meng Jiang1 1University of Notre Dame, Notre Dame, IN 2Microsoft Cognitive Services Research, Redmond, WA 3Carnegie Mellon University, Pittsburgh, PA 1{wyu1, mjiang2}@nd.edu; 2chezhu@microsoft.com Abstract Pre-trained language models (PLMs) aim to learn universal language representations by con- ducting self-supervised training tasks on large- scale corpus. Since PLMs capture word seman- tics in different contexts, the quality of word representations highly depends on word fre- quency, which usually follows a heavy-tailed distribution in the pre-training corpus. Thus, the embeddings of rare words on the tail are usually poorly optimized. In this work, we fo- cus on enhancing language model pre-training by leveraging definitions of the rare words in dictionary. To incorporate a rare word defini- tion as a part of input, we fetch it from the dictionary and append it to the end of the in- put text sequence. In addition to training with the masked language modeling objective, we propose two novel self-supervised pre-training tasks on word-level and sentence-level align- ment between the input text and rare word defi- nition to enhance language representations. We evaluate the proposed model named Dict-BERT on the GLUE benchmark and eight specialized domain datasets. Extensive experiments show that Dict-BERT significantly improves the un- derstanding of rare words and boosts model per- formance on various NLP downstream tasks. 1 Introduction Recently pre-trained language models (PLMs) such as BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) have revolutionized the field of natural language processing (NLP), yielding remarkable performance on various downstream tasks (Qiu et al., 2020). However, these PLMs suffer from lacking knowledge when completing real-world tasks. To address this issue, some methods have incorporated the knowledge to enrich language rep- resentations, ranging from linguistic (Wang et al., * This work was done when Wenhao Yu and Donghan Yu interned at Microsoft Cognitive Services Research group. 2021a), commonsense (Guan et al., 2020; Liu et al., 2020), factual (Wang et al., 2021b), to domain knowledge (Liu et al., 2020; Yu et al., 2022b). Nevertheless, rare words (Schick and Schütze, 2020) and unseen words (Cui et al., 2021) are still blind spots of pre-trained language models when they are fine-tuned on downstream tasks. For instance, in a dialogue system, users often talk to chatbots about recent hot topics, e.g., “Covid- 19”, which may not appear in the pre-training cor- pus (Cui et al., 2021). Since PLMs capture word semantics in different contexts to address the is- sue of polysemous and the context-dependent na- ture of words, consequently they usually perform poorly when a user mentions such novel words (Wu et al., 2021; Ruzzetti et al., 2021). As indicated by Wu et al. (2021), the quality of word represen- tations highly depends on the word frequency in the pre-training corpus, which typically follows a heavy-tail …

Content truncated to 500 words