VoxML: A Visualization Modeling Language

VoxML: A Visualization Modeling Language James Pustejovsky and Nikhil Krishnaswamy Brandeis University Waltham, Massachusetts USA {jamesp,nkrishna}@cs.brandeis.edu Abstract We present the specification for a modeling language, VoxML, which encodes semantic knowledge of real-world objects represented as three-dimensional models, and of events and attributes related to and enacted over these objects.VoxML is intended to overcome the limitations of existing 3D visual markup languages by allowing for the encoding of a broad range of semantic knowledge that can be exploited by a variety of systems and platforms, leading to multimodal simulations of real-world scenarios using conceptual objects that represent their semantic values. Keywords: Semantics, Cognitive Methods, Simulation, Lexical Semantics, Visualization 1. Introduction In this paper, we describe a modeling language for con- structing 3D visualizations of concepts denoted by natu- ral language expressions. This language, VoxML (Visual Object Concept Modeling Language), is being used as the platform for creating multimodal semantic simulations in the context of human-computer communication.1 Prior work in visualization from natural language has largely focused on object placement and orientation in static scenes (Coyne and Sproat, 2001; Siskind, 2001; Chang et al., 2015), and we have endeavored to incorporate dynamic semantics and motion language into our model. In previous work (Pustejovsky and Krishnaswamy, 2014; Pustejovsky, 2013), we introduced a method for modeling natural language expressions within a 3D simulation en- vironment built on top of the game development platform Unity (Goldstone, 2009). The goal of that work was to eval- uate, through explicit visualizations of linguistic input, the semantic presuppositions inherent in the different lexical choices of an utterance. This work led to two additional lines of research: an explicit encoding for how an object is itself situated relative to its environment; and an opera- tional characterization of how an object changes its location or how an agent acts on an object over time. The former has developed into a semantic notion of situational context, called a habitat (Pustejovsky, 2013; McDonald and Puste- jovsky, 2014), while the latter is addressed by dynamic in- terpretations of event structure (Pustejovsky and Moszkow- icz, 2011; Pustejovsky, 2013). The requirements on a visual simulation include, but are not limited to, the following components: 1. A minimal embedding space (MES) for the simula- tion must be determined. This is the 3D region within which the state is configured or the event unfolds; 2. Object-based attributes for participants in a situation or event need to be specified; e.g., orientation, relative size, default position or pose, etc.; 1This work is being carried out in the context of CwC, a DARPA effort to identify and construct computational semantic elements, for the purpose of carrying out joint plans between a human and computer through NL discourse. 3. An epistemic condition on the object and event ren- dering, imposing an implicit point of view (POV); 4. Agent-dependent embodiment; this determines the relative scaling of an agent and its event participants and their surroundings, as it engages in the environ- ment. In order to construct a robust simulation from linguistic in- …

Content truncated to 500 words