Hidden Schema Networks

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 4764–4798 July 9-14, 2023 ©2023 Association for Computational Linguistics Hidden Schema Networks Ramsés J. Sánchez1,2*, Lukas Conrads1,2, Pascal Welke1,2,3, Kostadin Cvejoski1,4, César Ojeda5 1Lamarr Institute, 2University of Bonn, 3TU Wien, 4Fraunhofer IAIS 5 University of Potsdam Abstract Large, pretrained language models infer pow- erful representations that encode rich seman- tic and syntactic content, albeit implicitly. In this work we introduce a novel neural language model that enforces, via inductive biases, ex- plicit relational structures which allow for com- positionality onto the output representations of pretrained language models. Specifically, the model encodes sentences into sequences of symbols (composed representations), which correspond to the nodes visited by biased ran- dom walkers on a global latent graph, and infers the posterior distribution of the latter. We first demonstrate that the model is able to uncover ground-truth graphs from artificially generated datasets of random token sequences. Next, we leverage pretrained BERT and GPT-2 language models as encoder and decoder, respectively, to infer networks of symbols (schemata) from nat- ural language datasets. Our experiments show that (i) the inferred symbols can be interpreted as encoding different aspects of language, as e.g. topics or sentiments, and that (ii) GPT-like models can effectively be conditioned on sym- bolic representations. Finally, we explore train- ing autoregressive, random walk “reasoning" models on schema networks inferred from com- monsense knowledge databases, and using the sampled paths to enhance the performance of pretrained language models on commonsense If-Then reasoning tasks. 1 Introduction Much of the developmental and causal theories of human cognition are predicated on relational structures of knowledge that naturally exhibit com- positionality. Semantic content is intrinsically re- lational, as one is only able to explain a given unit of knowledge – such as a concept, word or percep- tion – insofar as there are other units of knowledge which relate to it (Block, 1986; Margolis and Lau- rence, 1999). Thus we can partially construe a concept through its relationships to other concepts (like when we say “a dog is an animal that barks”), just as we can partially construe it through its rela- tionships to our perceptions (when we say “that is a dog”, whilst pointing to a dog on the street) or the words we use (when we use the word dog to refer to the concept dog). Likewise, we can partially construe words not only through their relationships to concepts or percepts, but also through their re- lationships to other words, as words that occur in the same context tend to have similar meanings (Harris, 1954; Firth, 1957). Note that is precisely this contextual semantic word content what we ex- plicit have access to, when processing our raw text datasets. On the other hand, generalization, reason- ing and understanding seem to be inevitably tied to the compositional nature of knowledge. Indeed, the ability to compose a set of knowledge units (and their relations) into new, more complex rela- tional units, …

Content truncated to 500 words