Bridging Perception, Memory, and Inference through Semantic Relations

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9136–9142 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 9136 Bridging Perception, Memory, and Inference through Semantic Relations∗ Johanna Björklund† Adam Dahlgren Lindström‡ Department of Computing Science Umeå University, Sweden {johanna,dali,drewes}@cs.umu.se Frank Drewes Abstract There is a growing consensus that surface form alone does not enable models to learn meaning and gain language understanding. This war- rants an interest in hybrid systems that combine the strengths of neural and symbolic methods. We favour triadic systems consisting of neural networks, knowledge bases, and inference en- gines. The network provides perception, that is, the interface between the system and its envi- ronment. The knowledge base provides explicit memory and thus immediate access to estab- lished facts. Finally, inference capabilities are provided by the inference engine which reflects on the perception, supported by memory, to reason and discover new facts. In this work, we probe six popular language models for seman- tic relations and outline a future line of research to study how the constituent subsystems can be jointly realised and integrated. 1 Introduction Recent works (Bender and Koller, 2020; Bender et al., 2021) postulate that it is impossible to learn meaning from surface form alone, and express con- cerns about what is perceived as an over-reliance on large-scale pretrained neural networks. This line of thought supports the interest in hybrid sys- tems that amalgamate elements from complemen- tary learning paradigms (see, e.g., (Pearl, 2019; Wang et al., 2019; Hohenecker and Lukasiewicz, 2020; van Bekkum et al., 2021)). In (Dahlgren et al., 2021), we argue that this calls for an explicit distinction to be made between the faculties of perception, memory, and inference. We therefore promote the development of systems that consist of subsystems with responsibilities corresponding to the three faculties. Such future systems would thus ∗ The authors are given in alphabetical order. † The work of this author is partially funded by the Swedish Research Council, Grant number 2020-03852, and the Wallenberg Autonomous Systems and AI program. ‡ Contact Author consist of a perception component realised by a neural network, a component that provides explicit memory in the form of a knowledge base, and a third one performing symbolic inference, that is, rule-based reasoning. We suggest to study how the subsystems can be aligned so for a seamless information flow be- tween them. We view it as particularly important that (i) the network and the knowledge base to- gether yield a consistent treatment of semantic re- lations and (ii) training takes the knowledge base into account, so that the resulting embeddings are consistent with established facts. Our conceptual discussion is complemented by a preliminary em- pirical evaluation of six popular English language models, which we subject to linear probes to test their abilities to capture central semantic relations. After a brief discussion of related work in Sec- tion 2, Section 3 discusses the role of semantic relations in the context of our envisioned triad sys- tem while Section …

Content truncated to 500 words