Solving Linguistic Olympiad Problems with Tree-of-Thought Prompting

Solving Linguistic Olympiad Problems with Tree-of-Thought Prompting Zheng-Lin Lin National Taiwan University b09208026@ntu.edu.tw Chiao-Han Yen National Taiwan University r11142008@ntu.edu.tw Jia-Cheng Xu National Taiwan University r07142006@ntu.edu.tw Deborah Watty National Taiwan University r11142012@ntu.edu.tw Shu-Kai Hsieh National Taiwan University shukaihsieh@ntu.edu.tw Abstract In this study, we delve into the efficacy of the Tree-of-Thought Prompting technique as a mechanism to address linguistic challenges and augment the reasoning capabilities of expansive language models. Specifically, we scrutinize the reasoning prowess of the Generative Pre-trained Transformer (GPT) model, which has garnered significant at- tention within the research and practitioner community. Utilizing the Tree-of-Thought Prompting methodology, we assess its utility in enhancing both the precision and response latency of the GPT model, especially for Linguistic Olympiad tasks demanding ele- vated reasoning competencies. Concurrently, we delineate inherent limitations within this approach and proffer avenues for future research to refine and optimize it. Code repo: https://github.com/chrizeroxtwo/ToT- LinguisticProblem Keywords: Tree-of-Thought Prompting, Large Language Models, Machine Reasoning, Gen- erative Pre-trained Transformer, Linguistic Olympiad 1 Introduction Large language models (LLMs) have experienced significant evolution, showcasing their versatile abilities in tackling a wide range of natural lan- guage processing (NLP) tasks. The Generative Pre-trained Transformer (GPT) model stands out as one of the most extensively discussed and influ- ential language models. By leveraging its founda- tion on large-scale text data pre-training, Liu et al. (2023) shows that GPT has given rise to numerous innovative applications across various domains. Among these tasks, its exceptional reasoning ability has emerged as a subject of fascination among researchers and practitioners. The adept- ness at proficient reasoning serves as a founda- tional element for various cognitive processes, shaping the intricate interplay between cognition and human capabilities. As such, understanding the underlying mechanisms of exceptional reason- ing holds substantial implications for cognitive psychology and related disciplines. To investigate the capacity for reasoning, a common area of fo- cus is complex problem-solving scenarios or log- ical reasoning tasks. Such subjects typically re- quire individuals to analyze intricate information, discern patterns, and draw well-structured conclu- sions from the available evidence. The selected tasks may encompass both deductive reasoning puzzles and inductive reasoning challenges, en- abling researchers to assess participants’ cognitive abilities in various contexts. The recently emerged research topic known as the “Rosetta Stone” problem addresses the afore- mentioned requirements effectively. This distinc- tive problem type involves the application of lim- ited data to “solve” and establish correspondences between expressions in two distinct language sys- tems (Bozhanov and Derzhanski, 2013). The Rosetta Stone task combines linguistic problems to create a general task that can be tackled by individuals without specialized linguis- tic skills. It encompasses a genre of composi- tion that presents linguistic facts and phenom- ena in an enigmatic form (Derzhanski and Payne, 2010). This eventually integrated into the Linguis- tic Olympiad (LO), akin to renowned competitions such as the United Kingdom Linguistics Olympiad (UKLO) 1 and the North American Computational Linguistics Open Competition (NACLO). 2 The Linguistic Olympiad (LO) encompasses various types of problems, focusing on different linguistic aspects such as …

Content truncated to 500 words