Enhancing Natural Language Representation with Large-Scale Out-of-Domain Commonsense

Findings of the Association for Computational Linguistics: ACL 2022, pages 1746 - 1756 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Enhancing Natural Language Representation with Large-Scale Out-of-Domain Commonsense Wanyun Cui, Xingran Chen Shanghai University of Finance and Economics cui.wanyun@sufe.edu.cn, xingran.chen.sufe@gmail.com Abstract We study how to enhance text representation via textual commonsense. We point out that commonsense has the nature of domain dis- crepancy. Namely, commonsense has different data formats and is domain-independent from the downstream task. This nature brings chal- lenges to introducing commonsense in general text understanding tasks. A typical method of introducing textual knowledge is continu- ing pre-training over the commonsense corpus. However, it will cause catastrophic forgetting to the downstream task due to the domain dis- crepancy. In addition, previous methods of di- rectly using textual descriptions as extra input information cannot apply to large-scale com- monsense. In this paper, we propose to use large-scale out-of-domain commonsense to enhance text representation. In order to effectively incor- porate the commonsense, we proposed OK- Transformer (Out-of-domain Knowledge en- hanced Transformer). OK-Transformer ef- fectively integrates commonsense descriptions and enhances them to the target text repre- sentation. In addition, OK-Transformer can adapt to the Transformer-based language mod- els (e.g. BERT, RoBERTa) for free, without pre-training on large-scale unsupervised cor- pora. We have verified the effectiveness of OK-Transformer in multiple applications such as commonsense reasoning, general text clas- sification, and low-resource commonsense set- tings. 1 1 Introduction Although unsupervised language models have achieved big success on many tasks (Devlin et al., 2019), they are incapable of learning low-frequency knowledge. For example, in the masked language model task in Fig. 1, even if we replace “Kevin was” (left) with “Jim was” (right), BERT (Devlin 1The code is available in https://github.com/ chenxran/ok-transformer et al., 2019) still predicts the masked word as sick, crying, dying, etc. This is because similar texts in its training corpus rarely describe the subject of “comforted”. To improve the model’s ability to generalize and understand low-frequency knowl- edge, we propose to incorporate commonsense into language models. In Fig. 1, to make correct predic- tions, we need to enhance the language model with the commonsense c1. However, commonsense has the nature of do- main discrepancy. The downstream task and the commonsense knowledge have distribution discrep- ancies. Taking the commonsense knowledge base we use (i.e. ATOMIC2020 (Hwang et al., 2020)) as an example, the distribution discrepancy is specif- ically manifested in (1) their data formats. The format of a commonsense description usually be- longs to some specific patterns (e.g. “... As a result ...”, “... Because ...”), while the downstream tasks can have arbitrary patterns. (2) The commonsense belongs to the domain of event causality, while the downstream tasks may belong to arbitrary domains. Here we highlight the challenges caused by the domain discrepancy. To introduce external tex- tual knowledge to a pre-trained language model, a common practice is to continue pre-training the lan- guage model on the corpus of the external knowl- edge (Gururangan et al., 2020; Sun et al., …

Content truncated to 500 words