Pretraining on Interactions for Learning Grounded Affordance Representations

Proceedings of the 11th Joint Conference on Lexical and Computational Semantics, pages 258 - 277 July 14-15, 2022 ©2022 Association for Computational Linguistics Pretraining on Interactions for Learning Grounded Affordance Representations Jack Merullo jack_merullo@brown.edu Dylan Ebert dylan_ebert@brown.edu Carsten Eickhoff carsten@brown.edu Ellie Pavlick ellie_pavlick@brown.edu Abstract Lexical semantics and cognitive science point to affordances (i.e. the actions that objects support) as critical for understanding and rep- resenting nouns and verbs. However, study of these semantic features has not yet been integrated with the “foundation” models that currently dominate language representation re- search. We hypothesize that predictive model- ing of object state over time will result in rep- resentations that encode object affordance in- formation “for free”. We train a neural network to predict objects’ trajectories in a simulated interaction and show that our network’s latent representations differentiate between both ob- served and unobserved affordances. We find that models trained using 3D simulations from our SPATIAL dataset outperform conventional 2D computer vision models trained on a similar task, and, on initial inspection, that differences between concepts correspond to expected fea- tures (e.g., roll entails rotation). Our results suggest a way in which modern deep learning approaches to grounded language learning can be integrated with traditional formal semantic notions of lexical representations. 1 Introduction Much of natural language semantics concerns events and their participants–i.e., verbs and the nouns with which they compose. Evidence from cognitive science (Borghi and Riggio, 2009; Maz- zuca et al., 2021) and neuroscience (Sakreida et al., 2013) suggests that grounding such words in per- ception is an essential part of linguistic process- ing, in particular suggesting that humans repre- sent nouns in terms of their affordances (Gib- son, 1977), i.e., the interactions which they sup- port. Affordance-based representations have been argued to form the basis of formal accounts of compositional syntax and semantics (Steedman, 2002). As such, prior work in formal semantics has sought to build grounded lexical semantic repre- sentations in terms of objects and their interactions Play with these objects! Seen Unseen Now play with this one! Can it roll? Can it contain other objects? Figure 1: We investigate whether observing interactions with an object in a 3D environment encodes information about their affordances and whether this generalizes in the zero shot setting to unseen object types in 3D space. For example, Pustejovsky and Kr- ishnaswamy (2014) and Siskind (2001) represent verbs like roll as a set of entailed positional and rotational changes specified in formal logic, and Pustejovsky and Krishnaswamy (2018) argue that nouns imply (latent) events–e.g., that cups gener- ally hold things–which should be encoded as TELIC values within the noun’s formal structure. Such work provides a compelling story of grounded semantics, but has not yet been connected to the types of large scale neural network models that currently dominate NLP. Thus, in this work, we ask whether such semantic representations emerge naturally as a consequence of self-supervised pre- dictive modeling. Our motivation stems from the success of predictive language modeling at encod- ing syntactic structure. …

Content truncated to 500 words