The VoxWorld Platform for Multimodal Embodied Agents

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 1529–1541 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 1529 The VoxWorld Platform for Multimodal Embodied Agents Nikhil Krishnaswamy∗, William Pickard∗, Brittany Cates∗, Nathaniel Blanchard∗, and James Pustejovsky† ∗Colorado State University †Brandeis University Fort Collins, CO, USA Waltham, MA, USA {nkrishna,william.pickard,brittany.cates, jamesp@brandeis.edu nathaniel.blanchard}@colostate.edu Abstract We present a five-year retrospective on the development of the VoxWorld platform, first introduced as a multimodal platform for modeling motion language, that has evolved into a platform for rapidly building and deploying embodied agents with contextual and situational awareness, capable of interacting with humans in multiple modalities, and exploring their environments. In particular, we discuss the evolution from the theoretical underpinnings of the VoxML modeling language to a platform that accommodates both neural and symbolic inputs to build agents capable of multimodal interaction and hybrid reasoning. We focus on three distinct agent implementations and the functionality needed to accommodate all of them: Diana, a virtual collaborative agent; Kirby, a mobile robot; and BabyBAW, an agent who self-guides its own exploration of the world. Keywords: multimodality, multimodal interaction, situated grounding, embodied agent, modeling platform, simulation 1. Introduction Multimodal language understanding has been a topic of intense interest in the natural language processing community over at least the last half a decade. The role that perception plays in human language under- standing has long been clear to much of the commu- nity and many efforts have attempted to build similar functionality into computer systems (Pylyshyn, 1978; Waltz, 1978; Rehm and Goecke, 2000; Rohlfing et al., 2003; Pecher and Zwaan, 2005; Mooney, 2008; Scherer et al., 2012; Krishnamurthy and Kollar, 2013; Miller and Johnson-Laird, 2013). There has also been interest in context-aware language understanding sys- tems, and the role that situatedness and embodiment play in such processes. This is an even harder prob- lem than merging computer vision and NLP into a sin- gle system, and perhaps because of this, interest has remained high. In 2016, at the LREC conference, we introduced a modeling language VoxML (Puste- jovsky and Krishnaswamy, 2016), intended to capture the semantics necessary to construct 3D visualizations of concepts denoted by linguistic expressions; as well as VoxSim, later that year at COLING 2016 (Krish- naswamy and Pustejovsky, 2016), a software system that generated such visualizations from the VoxML se- mantics. Interest in this approach has persisted through the com- munity (Cohen, 2017; Quick and Morrison, 2017; Fis- cher et al., 2018; Abrami et al., 2020; Bonn et al., 2020; Henlein et al., 2020; Rodrigues et al., 2020; Tamari et al., 2020; Kozierok et al., 2021; Richard-Bollans, 2021), and out of our initial goal to model visualized and situated object and event semantics grew an addi- tional line of research: creating intelligent agent behav- iors capable of performing reasoning over such seman- tics (Krishnaswamy et al., 2017; Narayana et al., 2018). VoxML and VoxSim became the platform on which to build such agent behaviors. However, intelligent agents …

Content truncated to 500 words