How much coffee was consumed during EMNLP 2019? Fermi Problems: A New Reasoning Challenge for AI

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7318–7328 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 7318 How Much Coffee Was Consumed During EMNLP 2019? Fermi Problems: A New Reasoning Challenge for AI Ashwin Kalyan1 Abhinav Kumar2 Arjun Chandrasekaran3 Ashish Sabharwal1 Peter Clark1 1Allen Institute for AI 2Georgia Tech 3Max Planck Institute for Intelligent Systems Abstract Many real-world problems require the com- bined application of multiple reasoning abilities—employing suitable abstractions, commonsense knowledge, and creative syn- thesis of problem-solving strategies. To help advance AI systems towards such capabilities, we propose a new reasoning challenge, namely Fermi Problems (FPs), which are questions whose answers can only be approximately estimated because their precise computation is either impractical or impossible. For example, “How much would the sea level rise if all ice in the world melted?” FPs are commonly used in quizzes and interviews to bring out and evaluate the creative reasoning abilities of humans. To do the same for AI systems, we present two datasets: 1) A collection of 1k real-world FPs sourced from quizzes and olympiads; and 2) a bank of 10k synthetic FPs of intermediate complexity to serve as a sandbox for the harder real-world challenge. In addition to question-answer pairs, the datasets contain detailed solutions in the form of an executable program and supporting facts, helping in supervision and evaluation of intermediate steps. We demonstrate that even extensively fine-tuned large-scale language models perform poorly on these datasets, on average making estimates that are off by two orders of magnitude. Our contribution is thus the crystallization of several unsolved AI problems into a single, new challenge that we hope will spur further advances in building systems that can reason. 1 Introduction How long is the drive from Seattle to NYC? How big of an emergency fund do I need? We frequently encounter such questions in our daily lives. Like- wise, scientists are often faced with questions such as, How much would the ocean surface rise if the ice caps melted? Known as Fermi Problems1 (FPs), 1after the celebrated physicist Enrico Fermi. See https: //en.wikipedia.org/wiki/Fermi_problem. these questions are problems whose answers can only be estimated within reasonable limits of er- ror, as precisely measuring the required quantity is either impossible or impractical. Solving a FP requires considerable life experi- ence, ability to think through long chains of reason- ing, and mathematical intuition – as illustrated by fig. 1 for a FP about rising sea levels. Answering an FP correctly requires multiple different facts, and the correct estimate can be arrived via various reasoning paths – this open-ended nature further adds to their challenge. Unsurprisingly, these ques- tions are often used to test candidates in science Olympiads and interviews. Due to the complex- ity of reasoning that is required to answer these questions, we propose solving FPs as a new task to drive progress in AI reasoning systems. A core skill required for solving FPs is that of estimation – e.g. “How much money do I need …

Content truncated to 500 words