Do PLMs Know and Understand Ontological Knowledge?

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 3080–3101 July 9-14, 2023 ©2023 Association for Computational Linguistics Do PLMs Know and Understand Ontological Knowledge? Weiqi Wu1, Chengyue Jiang1,2, Yong Jiang3∗, Pengjun Xie3, Kewei Tu1,2∗ 1School of Information Science and Technology, ShanghaiTech University 2Shanghai Engineering Research Center of Intelligent Vision and Imaging 3DAMO Academy, Alibaba Group, China {wuwq,jiangchy,tukw}@shanghaitech.edu.cn {yongjiang.jy,chengchen.xpj}@alibaba-inc.com Abstract Ontological knowledge, which comprises classes and properties and their relationships, is integral to world knowledge. It is significant to explore whether Pretrained Language Models (PLMs) know and understand such knowledge. However, existing PLM-probing studies focus mainly on factual knowledge, lacking a system- atic probing of ontological knowledge. In this paper, we focus on probing whether PLMs store ontological knowledge and have a semantic un- derstanding of the knowledge rather than rote memorization of the surface form. To probe whether PLMs know ontological knowledge, we investigate how well PLMs memorize: (1) types of entities; (2) hierarchical relationships among classes and properties, e.g., Person is a subclass of Animal and Member of Sports Team is a subproperty of Member of; (3) domain and range constraints of properties, e.g., the subject of Member of Sports Team should be a Person and the object should be a Sports Team. To further probe whether PLMs truly understand ontological knowledge beyond memorization, we comprehensively study whether they can reliably perform logical reasoning with given knowledge according to ontological entailment rules. Our probing results show that PLMs can memorize certain ontological knowledge and utilize implicit knowledge in reasoning. How- ever, both the memorizing and reasoning per- formances are less than perfect, indicating in- complete knowledge and understanding. 1 Introduction Pretrained Language Models (PLMs) have or- chestrated impressive progress in NLP across a wide variety of downstream tasks, including knowledge-intensive tasks. Previous works pro- pose that PLMs are capable of encoding a signif- icant amount of knowledge from the pretraining corpora (AlKhamissi et al., 2022), and determine to explore the kinds of knowledge within PLMs. ∗Yong Jiang and Kewei Tu are corresponding authors. Reasoning Manual: Rule rdfs2 Manual: domain I. Explicitly Given II. Implicitly Given III. Not Given (b) Memorizing person person organisation organisation Premise 1/2 One has to be a particular [MASK] to be a player at a sports team. … … z Member of sports team [s][s][s] [MASK] . Soft: domain Pseudoword (P1) One has to be a person to be a player at a sports team. (P2) Lionel Messi is a player at Argentina National Football Team. (H) Therefore, Lionel Messi is a [MASK] . Member of Sports Team Person Member of … Sports Team Animal … Lionel Messi Argentina National Football Team … subclass of subproperty of domain range type type (a) Member of Sports Team subclass of subclass of subclass of subproperty of Organization Figure 1: (a) An example of an ontological knowledge graph. (b) Potential manual and soft prompts to probe the knowledge and corresponding semantics. Instances are replaced by pseudowords in reasoning experiments …

Content truncated to 500 words