PROST: Physical Reasoning about Objects through Space and Time

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4597–4608 August 1–6, 2021. ©2021 Association for Computational Linguistics 4597 PROST : Physical Reasoning about Objects through Space and Time Stéphane Aroca-Ouellette∗, Cory Paik, Alessandro Roncone and Katharina Kann University of Colorado Boulder firstname.lastname@colorado.edu Abstract We present a new probing dataset named PROST: Physical Reasoning about Objects Through Space and Time. This dataset con- tains 18,736 multiple-choice questions made from 14 manually curated templates, covering 10 physical reasoning concepts. All questions are designed to probe both causal and masked language models in a zero-shot setting. We conduct an extensive analysis which demon- strates that state-of-the-art pretrained models are inadequate at physical reasoning: they are influenced by the order in which answer op- tions are presented to them, they struggle when the superlative in a question is inverted (e.g., most ↔ least), and increasing the amount of pretraining data and parameters only yields minimal improvements. These results provide support for the hypothesis that current pre- trained models’ ability to reason about physi- cal interactions is inherently limited by a lack of real world experience. By highlighting these limitations, we hope to motivate the de- velopment of models with a human-like under- standing of the physical world. 1 Introduction In the context of natural language processing (NLP), Bender and Koller (2020) provides a working defi- nition of “understanding” as the ability to recover the communicative intent from an utterance. To achieve this, one must be able to query a set of concepts that is aligned with the speaker’s own un- derstanding. An example of such alignment is our interaction with the physical world. This experi- ence, shared by all humans, provides a common set of concepts to rely on in communication. For example, the reader can map the phrase I dropped my pint glass to a set of relevant experiences and generate a mental depiction of the scene. Further, ∗*Email has no accent, but includes the hyphen A) glass B) pillow C) coin D) pen A person drops a glass, a pillow, a coin, and a pen from a balcony. The [MASK] is most likely to break. Figure 1: An example question from PROST. the reader can also use their knowledge of gravity and the properties of a pint glass to reason about potential outcomes: the pint glass will fall toward the ground and will likely break on impact. Children grab, push, and play with the objects around them to form concepts about the world they live in even before learning to talk (Hespos and Spelke, 2004). These concepts are then linked with words to enable communication, eventually pro- viding the necessary grounds for concepts and lan- guage to co-develop (Bloom, 2002; Gelman, 2009). In contrast, current language models (LMs) are not exposed to real-world experiences, making them in- capable of grounding language (Bisk et al., 2020a). We hypothesize that this lack of experience im- pedes their ability to both understand an utterance relating to the physical world and their ability to …

Content truncated to 500 words