KACC: A Multi-task Benchmark for Knowledge Abstraction, Concretization and Completion

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1751–1763 August 1–6, 2021. ©2021 Association for Computational Linguistics 1751 KACC: A Multi-task Benchmark for Knowledge Abstraction, Concretization and Completion Jie Zhou1,2∗, Shengding Hu1,2∗, Xin Lv1,2, Cheng Yang4, Zhiyuan Liu1,2,3†, Wei Xu5, Jie Jiang5, Juanzi Li1,2, Maosong Sun1,2,3 1Department of Computer Science and Technology 2Institute for Artificial Intelligence 3State Key Lab on Intelligent Technology and Systems, Tsinghua University, Beijing, China 4School of Computer Science, Beijing University of Posts and Telecommunications, China 5Tencent Marketing Solution, Tencent, Shenzhen, China {zhoujie18, hsd20, lv-x18}@mails.tsinghua.edu.cn Abstract A comprehensive knowledge graph (KG) con- tains an instance-level entity graph and an ontology-level concept graph. The two-view KG provides a testbed for models to “simu- late” human’s abilities on knowledge abstrac- tion, concretization, and completion (KACC), which are crucial for human to recognize the world and manage learned knowledge. Ex- isting studies mainly focus on partial aspects of KACC. In order to promote thorough anal- yses for KACC abilities of models, we pro- pose a unified KG benchmark by improving existing benchmarks in terms of dataset scale, task coverage, and difficulty. Specifically, we collect new datasets that contain larger concept graphs, abundant cross-view links as well as dense entity graphs. Based on the datasets, we propose novel tasks such as multi- hop knowledge abstraction (MKA), multi-hop knowledge concretization (MKC) and then de- sign a comprehensive benchmark. For MKA and MKC tasks, we further annotate multi-hop hierarchical triples as harder samples. The ex- perimental results of existing methods demon- strate the challenges of our benchmark. The re- source is available at https://github.com/ thunlp/KACC. 1 Introduction Large-scale knowledge graphs (KGs) like Wiki- data (Vrandeˇci´c and Kr¨otzsch, 2014), DBpe- dia (Lehmann et al., 2015), and YAGO (Mahdis- oltani et al., 2013) usually contain two subgraphs: an instance-level entity graph and an ontology- level concept graph. The entity graph (a.k.a. the entity view) is composed of entities and relations. It describes factual knowledge such as (Da Vinci, paint, Mona Lisa). The concept graph (a.k.a. ∗ indicates equal contribution † Corresponding author: Z.Liu(liuzy@tsinghua.edu.cn) �������� ��������� ����� ����� ������� ��������������� ������� �������� ������� ������ ����������� ����������� ������� ����� ������ ����� ������ ����������� ������ �������� ���� ������ �������� ����� ����������� ����������� ��������� ����������� ������������������ ����������������������������� ����������� ����������� Figure 1: An example of the entity-concept KG. the concept view) contains concepts and concep- tual relations. It provides abstract and common- sense knowledge like (painter, create, paint- ing). In this paper, we name this kind of two-view KG as the entity-concept KG (EC-KG). In a EC- KG, the relations can be grouped into three cate- gories. The “subclassOf” relation forms hierar- chical concept structures via triples like (painter, subclassOf, artist). The “instanceOf” re- lation groups entities into concepts, such as (Da Vinci, instanceOf, painter). These two rela- tions are important for testing models’ abilities on knowledge abstraction and concretization. Other relations are logical relations for testing models’ abilities on knowledge completion. An example of the EC-KG is shown in Figure 1. During the last decade, there are …

Content truncated to 500 words