Benchmarking Commonsense Knowledge Base Population with an Effective Evaluation Dataset

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8949–8964 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 8949 Benchmarking Commonsense Knowledge Base Population with an Effective Evaluation Dataset Tianqing Fang1∗, Weiqi Wang1∗, Sehyun Choi1, Shibo Hao2, Hongming Zhang1, Yangqiu Song1, & Bin He3 1Department of Computer Science and Engineering, HKUST 2School of Electronics Engineering and Computer Science, Peking University 3Huawei Noah’s Ark Lab tfangaa@cse.ust.hk, {wwangbw,schoiaj}@connect.ust.hk, haoshibo@pku.edu.cn, {hzhangal,yqsong}@cse.ust.hk, hebin.nlp@huawei.com Abstract Reasoning over commonsense knowledge bases (CSKBs) whose elements are in the form of free-text is an important yet hard task in NLP. While CSKB completion only fills the missing links within the domain of the CSKB, CSKB population is alternatively pro- posed with the goal of reasoning unseen as- sertions from external resources. In this task, CSKBs are grounded to a large-scale eventual- ity (activity, state, and event) graph to discrimi- nate whether novel triples from the eventuality graph are plausible or not. However, existing evaluations on the population task are either not accurate (automatic evaluation with ran- domly sampled negative examples) or of small scale (human annotation). In this paper, we benchmark the CSKB population task with a new large-scale dataset by first aligning four popular CSKBs, and then presenting a high- quality human-annotated evaluation set to probe neural models’ commonsense reasoning ability. We also propose a novel inductive com- monsense reasoning model that reasons over graphs. Experimental results show that gen- eralizing commonsense reasoning on unseen assertions is inherently a hard task. Models achieving high accuracy during training per- form poorly on the evaluation set, with a large gap between human performance. Codes and data are available at https://github.com/ HKUST-KnowComp/CSKB-Population. 1 Introduction Commonsense reasoning is one of the core prob- lems in the field of artificial intelligence. Through- out the development in computational common- sense, commonsense knowledge bases (CSKB) (Speer et al., 2017; Sap et al., 2019) are constructed to enhance models’ reasoning ability. As human- annotated CSKBs are far from complete due to ∗ Equal Contribution Figure 1: Comparison between CSKB completion and population. An example of aligning eventuality graph as candidate commonsense knowledge triples is also provided. the scale of crowd-sourcing, reasoning tasks such as CSKB completion (Li et al., 2016; Malaviya et al., 2020; Moghimifar et al., 2021) and popu- lation (Fang et al., 2021) are proposed to enrich the missing facts. The CSKB completion task is defined based on the setting of predicting missing links within the CSKB. On the other hand, the pop- ulation task grounds commonsense knowledge in CSKBs to large-scale automatically extracted can- didates, and requires models to determine whether a candidate triple, (head, relation, tail), is plausi- ble or not, based on the information from both the CSKB and the large number of candidates which essentially form a large-scale graph structure. An illustration of the difference between completion and population is shown in Figure 1. There are two advantages of the population task. First, the population can not only add links but also nodes to an …

Content truncated to 500 words