ScienceWorld: Is your Agent Smarter than a 5th Grader?

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11279 - 11298 December 7-11, 2022 ©2022 Association for Computational Linguistics SCIENCEWORLD: Is your Agent Smarter than a 5th Grader? Ruoyao Wang∗‡, Peter Jansen∗‡, Marc-Alexandre Côté♣, Prithviraj Ammanabrolu♢ ‡University of Arizona, Tucson, AZ ♣Microsoft Research Montréal ♢Allen Institute for AI, Seattle, WA {ruoyaowang,pajansen}@arizona.edu macote@microsoft.com, raja@allenai.org Abstract We present SCIENCEWORLD, a benchmark to test agents’ scientific reasoning abilities in a new interactive text environment at the level of a standard elementary school science curricu- lum. Despite the transformer-based progress seen in question-answering and scientific text processing, we find that current models cannot reason about or explain learned science con- cepts in novel contexts. For instance, mod- els can easily answer what the conductivity of a known material is but struggle when asked how they would conduct an experiment in a grounded environment to find the conductivity of an unknown material. This begs the question of whether current models are simply retriev- ing answers by way of seeing a large number of similar examples or if they have learned to rea- son about concepts in a reusable manner. We hypothesize that agents need to be grounded in interactive environments to achieve such rea- soning capabilities. Our experiments provide empirical evidence supporting this hypothesis— showing that a 1.5 million parameter agent trained interactively for 100k steps outperforms a 11 billion parameter model statically trained for scientific question-answering and reasoning from millions of expert demonstrations.12 1 Introduction Question answering (QA) has seen rapid progress recently. Standardized elementary and middle school science exams have served as a challenge task for QA (Clark et al., 2018), as these questions require combining science-domain knowledge with world knowledge in complex reasoning procedures to solve. As large language models have toppled these benchmarks (Clark et al., 2020; Khashabi et al., 2020; Xu et al., 2021a), the focus has shifted away from simply answering questions toward pro- ducing human-readable explanations for a model’s 1Website: https://sciworld.apps.allenai.org 2Code: https://github.com/allenai/ScienceWorld SCIENCEWORLD Task 3 (Test Electrical Conductivity Subtask) Task Description Your task is to determine if a metal fork is electrically conductive. The metal fork is located around the kitchen. First, focus on the metal fork. If conductive, place it in the blue box. If nonconductive, place it in the green box. Agent starts in a random location, navigates to kitchen Kitchen This room is called the kitchen. In it, you see: a fridge. The freezer door is closed. a cupboard. The cupboard door is closed. a table. On the table is: a glass cup (containing nothing). a stove, which is turned of. On the stove is: nothing. a sink, which is turned of. In the sink is: nothing. a thermometer, currently reading a temperature of 10 degrees celsius a metal fork a counter. On the counter is: a drawer, a bowl (containing an orange, a banana, a potato, a red apple). You also see: A door to the outside (that is open) A door to the bathroom (that is open) …

Content truncated to 500 words