Do Trajectories Encode Verb Meaning?

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2860 - 2871 July 10-15, 2022 ©2022 Association for Computational Linguistics Do Trajectories Encode Verb Meaning? Dylan Ebert Brown University dylan_ebert@brown.edu Chen Sun Brown University chensun@brown.edu Ellie Pavlick Brown University ellie_pavlick@brown.edu Abstract Distributional models learn representations of words from text, but are criticized for their lack of grounding, or the linking of text to the non- linguistic world. Grounded language models have had success in learning to connect con- crete categories like nouns and adjectives to the world via images and videos, but can strug- gle to isolate the meaning of the verbs them- selves from the context in which they typically occur. In this paper, we investigate the extent to which trajectories (i.e. the position and ro- tation of objects over time) naturally encode verb semantics. We build a procedurally gen- erated agent-object-interaction dataset, obtain human annotations for the verbs that occur in this data, and compare several methods for rep- resentation learning given the trajectories. We find that trajectories correlate as-is with some verbs (e.g., fall), and that additional abstrac- tion via self-supervised pretraining can further capture nuanced differences in verb meaning (e.g., roll vs. slide). 1 Introduction While large distributional language models such as BERT (Devlin et al., 2019) and GPT (Radford, 2020; Brown et al., 2020) have had empirical suc- cess in deriving representations of words and sen- tences from large text corpora, most of these mod- els lack grounding, or a connection between the words and their real-world referents. Grounding, in addition to being necessary for multimodal tasks like video recognition, has been argued to lie at the core of language understanding (Bender and Koller, 2020). Work on grounded language learning asso- ciates language with the non-linguistic world, typ- ically by learning from large-scale image (Bruni et al., 2011) or video (Sun et al., 2019) datasets. Much prior work on language grounding has fo- cused on concrete nouns (objects) and adjectives (attributes), which are captured well by patterns of pixels. Verbs, however, have received less attention, despite being essential for building models that can interact in realistic 3D environments (Shridhar et al., 2020a; Bisk et al., 2020). Verbs are espe- cially challenging to model, given that they take place over time. Image and video data alone is in- sufficient to fully capture verb semantics, as demon- strated by prior work (Yatskar et al., 2016), in many cases failing to isolate the meaning of the verb from context in which it typically occurs. For example, Chao et al. 2018 show that an image of a person laying in the snow next to a snowboard is labeled “standing on a snowboard". Moreover, recent work has introduced datasets and benchmarks based on situated 3D environments (Gan et al., 2020; Deitke et al., 2020; Ebert and Pavlick, 2020; Shridhar et al., 2020a) that demonstrate the challenges of learning task-oriented behavior, which demands a combina- tion of object and verb …

Content truncated to 500 words