CoCoLM: Complex Commonsense Enhanced Language Model with Discourse Relations

Findings of the Association for Computational Linguistics: ACL 2022, pages 1175 - 1187 May 22-27, 2022 c⃝2022 Association for Computational Linguistics CoCoLM: Complex Commonsense Enhanced Language Model with Discourse Relations Changlong Yu1∗ Hongming Zhang1, 2∗ Yangqiu Song1 Wilfred Ng1 1HKUST, Hong Kong, China 2Tencent AI Lab, Seattle, U.S. {cyuaq, yqsong, wilfred}@cse.ust.hk, hongmzhang@tencent.com Abstract Large-scale pre-trained language models have demonstrated strong knowledge representa- tion ability. However, recent studies suggest that even though these giant models contain rich simple commonsense knowledge (e.g., bird can fly and fish can swim.), they often struggle with complex commonsense knowl- edge that involves multiple eventualities (verb- centric phrases, e.g., identifying the relation- ship between “Jim yells at Bob” and “Bob is upset”). To address this issue, in this paper, we propose to help pre-trained lan- guage models better incorporate complex com- monsense knowledge. Unlike direct fine- tuning approaches, we do not focus on a spe- cific task and instead propose a general lan- guage model named CoCoLM. Through the careful training over a large-scale eventual- ity knowledge graph ASER, we successfully teach pre-trained language models (i.e., BERT and RoBERTa) rich discourse-level common- sense knowledge among eventualities. Experi- ments on multiple commonsense tasks that re- quire the correct understanding of eventuali- ties demonstrate the effectiveness of CoCoLM. 1 Introduction Recently, large-scale pre-trained language repre- sentation models (LMs) (Devlin et al., 2019; Liu et al., 2019) have demonstrated the strong ability to discover useful linguistic properties of syntax and remember an impressive amount of knowledge with self-supervised training over a large unlabeled corpus (Petroni et al., 2019; Jiang et al., 2020). On top of that, with the help of the fine-tuning step, LMs can learn how to use the memorized knowledge for different tasks, and thus achieve out- standing performance on many downstream natural language processing (NLP) tasks. As discussed in Verga et al. (2020), while lan- guage models have already captured rich knowl- ∗ Equal contribution. Query Answer Birds can [MASK]. fly Cars are used for [MASK]. transport Jim yells at Bob, [MASK] Jim is upset. but Jim yells at Bob, [MASK] Bob is upset. but Table 1: Exploring knowledge contained in pre-trained language models following LAMA (Petroni et al., 2019). Queries and prediction returned by BERT-large are presented. Semantically plausible and implausible prediction are indicated with blue and red colors. edge, they often only perform well when the se- mantic unit is a single token while poorly when the semantic unit is more complex (e.g., a multi-token named entity or an eventuality, which is a linguis- tic term for verb-centric phrases covering activities, states and events (Bach, 1986; Araki and Mitamura, 2018)). For example, as shown in Table 1, if we follow LAMA (Petroni et al., 2019) to analyze the knowledge contained in BERT-large (Devlin et al., 2019) with a token prediction task, we can find out that BERT can understand that birds can fly, and a car is used for transportation, but it fails to under- stand the relation between “Jim yells at Bob” and …

Content truncated to 500 words