ESPRIT: Explaining Solutions to Physical Reasoning Tasks

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7906–7917 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 7906 ESPRIT: Explaining Solutions to Physical Reasoning Tasks Nazneen Fatema Rajani1∗ Rui Zhang2∗ Yi Chern Tan2 Stephan Zheng1 Jeremy Weiss2 Aadit Vyas2 Abhijit Gupta2 Caiming Xiong1 Richard Socher1 Dragomir Radev1,2 ∗Equal contribution. 1 Salesforce Research 2 Yale University {nazneen.rajani, stephan.zheng, dradev}@salesforce.com {r.zhang, yichern.tan, dragomir.radev}@yale.edu Abstract Neural networks lack the ability to reason about qualitative physics and so cannot gen- eralize to scenarios and tasks unseen during training. We propose ESPRIT, a framework for commonsense reasoning about qualitative physics in natural language that generates in- terpretable descriptions of physical events. We use a two-step approach of first identifying the pivotal physical events in an environment and then generating natural language descrip- tions of those events using a data-to-text ap- proach. Our framework learns to generate explanations of how the physical simulation will causally evolve so that an agent or a hu- man can easily reason about a solution using those interpretable descriptions. Human eval- uations indicate that ESPRIT produces crucial fine-grained details and has high coverage of physical concepts compared to even human annotations. Dataset, code and documenta- tion are available at https://github.com/ salesforce/esprit. 1 Introduction Humans learn to understand and reason about phys- ical laws just by living in this world and doing everyday things. AI models, on the other hand, lack this ability and so are unable to generalize to new scenarios that require reasoning about ab- stract physical concepts like gravity, mass, iner- tia, friction, and collisions (Bakhtin et al., 2019). We propose Explaining Solutions to Physical Rea- sonIng Tasks (ESPRIT), a framework for explain- ing qualitative physics reasoning using natural lan- guage. Neural networks with knowledge of qualita- tive physics would have commonsense reasoning abilities about the way the world works (Forbus, 1988). In turn, this could, for example, improve performance on tasks that involve interacting with humans and make human-robot interactions more efficient and trustworthy. Figure 1: An example from the PHYRE dataset (Bakhtin et al., 2019) consisting of a goal, an initial scene, a solution – the action of adding a red ball, and the resulting simulation rollout. Each object color cor- responds to an object type. Red: user-added dynamic object; Green and Blue: dynamic goal object; Gray: dynamic scene object; Black: static scene object. Ideally, AI systems would reason about and gen- erate natural language commonsense explanations of physical concepts that are relevant to their behav- ior and prediction. A key intuition is that natural language can provide an efficient low-dimensional representation of complicated physical concepts. To equip AI systems with this ability, we collected a set of open-ended natural language human expla- nations of qualitative physics simulations. The ex- planations include descriptions of the initial scene, i.e., before any physics is at play, and a sequence of identified pivotal events in a physics simula- tion. Three physical concepts are crucial for our simulation to reach a specified goal state: …

Content truncated to 500 words