True Detective: A Deep Abductive Reasoning Benchmark Undoable for GPT-3 and Challenging for GPT-4

Proceedings of the The 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023), pages 314–322 July 13-14, 2023 ©2023 Association for Computational Linguistics True Detective: A Deep Abductive Reasoning Benchmark Undoable for GPT-3 and Challenging for GPT-4 Maksym Del and Mark Fishel Institute of Computer Science University of Tartu, Estonia {maksym,mark}@tartunlp.ai Abstract Large language models (LLMs) have demon- strated solid zero-shot reasoning capabilities, which is reflected in their performance on the current test tasks. This calls for a more chal- lenging benchmark requiring highly advanced reasoning ability to be solved. In this paper, we introduce such a benchmark, consisting of 191 long-form (1200 words on average) mys- tery narratives constructed as detective puzzles. Puzzles are sourced from the "5 Minute Mys- tery" platform and include a multiple-choice question for evaluation. Only 47% of humans solve a puzzle successfully on average, while the best human solvers achieve over 80% suc- cess rate. We show that GPT-3 models barely outperform random on this benchmark (with 28% accuracy) while state-of-the-art GPT-4 solves only 38% of puzzles. This indicates that there is still a significant gap in the deep reasoning abilities of LLMs and humans and highlights the need for further research in this area. Our work introduces a challenging bench- mark for future studies on reasoning in lan- guage models and contributes to a better under- standing of the limits of LLMs’ abilities.1 1 Introduction Large language models (LLMs) have gained signifi- cant attention in recent years due to their impressive performance on a wide range of natural language processing tasks, including reasoning tasks (Srivas- tava et al., 2022; Wei et al., 2022). This calls for new, genuinely challenging benchmarks requiring LLMs to possess truly advanced reasoning capabil- ities to be solved. Abductive reasoning is a type of inference aim- ing at finding the minimal and most justified ex- planation for the set of phenomena or observa- tions. Previous benchmarks on this topic, such as Mostafazadeh et al. (2016), consisted of short 1https://github.com/TartuNLP/ true-detective and straightforward common-sense observations and were solved by GPT models (Radford and Narasimhan, 2018). However, the canonical exam- ple of abductive reasoning, a demanding process of a detective finding the best solution to a complex crime based on the clues and observations, was not explored as a foundation for the LLM benchmark in the literature. Motivated by the need for a new reasoning benchmark and inspired by the complexities and particularities of a detective enterprise, we present a novel abductive reasoning benchmark consist- ing of 191 detective puzzles/mysteries. Mysteries are sourced from the "5 Minute Mystery" platform, where professional and aspiring authors wrote them. A puzzle is structured as a >1000 words story with 4-5 answer options. Over the last 15 years, puzzles were attempted by humans around 2000 times each with an average solve rate of 47% (only the first try for each human for each puzzle counts). However, top human solvers (top 10) achieve a success rate of over 80% solving more than 154 of 192 …

Content truncated to 500 words