On the Generalization Ability of Retrieval-Enhanced Transformers

Findings of the Association for Computational Linguistics: EACL 2023, pages 133–148 May 2-6, 2023 ©2023 Association for Computational Linguistics Learning the Effects of Physical Actions in a Multi-modal Environment Gautier Dagan, Frank Keller, Alex Lascarides School of Informatics University of Edinburgh, UK gautier.dagan@ed.ac.uk, {keller, alex}@inf.ed.ac.uk Abstract Large Language Models (LLMs) handle phys- ical commonsense information inadequately. As a result of being trained in a disembodied setting, LLMs often fail to predict an action’s outcome in a given environment. However, predicting the effects of an action before it is executed is crucial in planning, where coher- ent sequences of actions are often needed to achieve a goal. Therefore, we introduce the multi-modal task of predicting the outcomes of actions solely from realistic sensory inputs (images and text). Next, we extend an LLM to model latent representations of objects to bet- ter predict action outcomes in an environment. We show that multi-modal models can capture physical commonsense when augmented with visual information. Finally, we evaluate our model’s performance on novel actions and ob- jects and find that combining modalities help models to generalize and learn physical com- monsense reasoning better. 1 Introduction Large Language Models (LLMs) are trained on large corpora of disembodied texts. They are typi- cally pre-trained on a masked language modeling task: the model must predict a masked word in a text given its context. LLMs have achieved state- of-the-art performance on many NLP tasks (Devlin et al., 2019; Brown et al., 2020), but they can also fail on seemingly easy and obvious tasks and in un- predictable ways (McCoy et al., 2020; Bommasani et al., 2021). Commonsense knowledge is shared knowledge and is often so obvious that it is absent from the LLMs’ training data: people don’t men- tion what is already known to their interlocutors. This includes physical commonsense information, including how executed actions affect the physical attributes of objects; e.g., shape and weight (Forbes et al., 2019). Humans may learn such knowledge from their embodied environment. But LLMs, be- ing trained on disembodied text, can make incorrect predictions about physical attributes and how these change when actions occur. For instance, when asked what the weight of a 150 grams potato after it is sliced, GPT-3 (Brown et al., 2020) incorrectly answers 75 grams (see Appendix A for the exact prompt). GPT-3 is an LLM with 175 billion param- eters, and nonetheless its disembodied existence limits its physical commonsense estimates. Zellers et al. (2021) inject physical common- sense information into LLMs via their model PIGLeT—a modified LLM that is trained on their PIGPeN simulated 3D environment dataset. PIGLeT estimates how an environment changes as a result of specific actions. In training and testing, the model uses ground-truth symbolic representa- tions of the environment but not the images: it ignores visual sensory observations. These sym- bolic representations of objects in an environment are chosen to capture the possible effects of ac- tions, and include attributes like weight, size and temperature. However, in an embodied situation, an agent needs …

Content truncated to 500 words