In-Context Analogical Reasoning with Pre-Trained Language Models

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 1953–1969 July 9-14, 2023 ©2023 Association for Computational Linguistics In-Context Analogical Reasoning with Pre-Trained Language Models Xiaoyang Hu12∗ Shane Storks1∗ Richard L. Lewis2† Joyce Chai1† 1Computer Science and Engineering Division, University of Michigan 2Department of Psychology, University of Michigan {nickhu, sstorks, rickl, chaijy}@umich.edu Abstract Analogical reasoning is a fundamental capacity of human cognition that allows us to reason abstractly about novel situations by relating them to past experiences. While it is thought to be essential for robust reasoning in AI sys- tems, conventional approaches require signif- icant training and/or hard-coding of domain knowledge to be applied to benchmark tasks. Inspired by cognitive science research that has found connections between human language and analogy-making, we explore the use of in- tuitive language-based abstractions to support analogy in AI systems. Specifically, we apply large pre-trained language models (PLMs) to visual Raven’s Progressive Matrices (RPM), a common relational reasoning test. By simply encoding the perceptual features of the problem into language form, we find that PLMs exhibit a striking capacity for zero-shot relational reason- ing, exceeding human performance and nearing supervised vision-based methods. We explore different encodings that vary the level of ab- straction over task features, finding that higher- level abstractions further strengthen PLMs’ ana- logical reasoning. Our detailed analysis reveals insights on the role of model complexity, in- context learning, and prior knowledge in solv- ing RPM tasks. 1 Introduction Humans are constantly presented with novel prob- lems and circumstances. Rather than understand them in isolation, we try to connect them with past experiences. With any luck, we might find an anal- ogy: a mapping between relevant aspects of this new situation and a past situation, which helps form abstractions that allow us to reason more ef- fectively in the future (Holyoak, 1984). Analogy is thought to underpin humans’ robust reasoning and problem solving capabilities (Hofstadter and ∗Authors contributed equally to this work. † Equal advising contribution. Language-Based Abstractions … … … … … … Pre-Trained Language Model Generated Prompts 8-Way Visual Raven’s Progressive Matrix (RPM) P( | ) Figure 1: Raven’s Progressive Matrices (Raven and Court, 1938; Zhang et al., 2019a) are an analogy-making task where one must infer the missing matrix item based on abstract rules instantiated in the first two rows. To demonstrate the potential analogical reasoning skills in pre-trained language models, we develop language- based abstractions over their key perceptual features, then prompt them to select the completion of the matrix. Sander, 2013), and thus it is believed to be prereq- uisite in order to enable the same in AI systems. However, conventional approaches struggle with analogy-making, and are trained on thousands of examples to achieve any success on benchmark tasks. This is unsatisfying, as humans are capa- ble of analogy-making without explicit training, and such analogy-making should enable zero-shot generalization to new situations (Mitchell, 2021). Interestingly, a body of work in cognitive sci- ence suggests that analogy-making and relational …

Content truncated to 500 words