Learning Spatial Knowledge for Text to 3D Scene Generation

Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2028–2038, October 25-29, 2014, Doha, Qatar. c⃝2014 Association for Computational Linguistics Learning Spatial Knowledge for Text to 3D Scene Generation Angel X. Chang, Manolis Savva and Christopher D. Manning Stanford University {angelx,msavva,manning}@cs.stanford.edu Abstract We address the grounding of natural lan- guage to concrete spatial constraints, and inference of implicit pragmatics in 3D en- vironments. We apply our approach to the task of text-to-3D scene generation. We present a representation for common sense spatial knowledge and an approach to ex- tract it from 3D scene data. In text-to- 3D scene generation, a user provides as in- put natural language text from which we extract explicit constraints on the objects that should appear in the scene. The main innovation of this work is to show how to augment these explicit constraints with learned spatial knowledge to infer missing objects and likely layouts for the objects in the scene. We demonstrate that spatial knowledge is useful for interpreting natu- ral language and show examples of learned knowledge and generated 3D scenes. 1 Introduction To understand language, we need an understanding of the world around us. Language describes the world and provides symbols with which we rep- resent meaning. Still, much knowledge about the world is so obvious that it is rarely explicitly stated. It is uncommon for people to state that chairs are usually on the floor and upright, and that you usu- ally eat a cake from a plate on a table. Knowledge of such common facts provides the context within which people communicate with language. There- fore, to create practical systems that can interact with the world and communicate with people, we need to leverage such knowledge to interpret lan- guage in context. Spatial knowledge is an important aspect of the world and is often not expressed explicitly in nat- ural language. This is one of the biggest chal- Figure 1: Generated scene for “There is a room with a chair and a computer.” Note that the system infers the presence of a desk and that the computer should be supported by the desk. lenges in grounding language and enabling natu- ral communication between people and intelligent systems. For instance, if we want a robot that can follow commands such as “bring me a piece of cake”, it needs to be imparted with an understand- ing of likely locations for the cake in the kitchen and that the cake should be placed on a plate. The pioneering WordsEye system (Coyne and Sproat, 2001) addressed the text-to-3D task and is an inspiration for our work. However, there are many remaining gaps in this broad area. Among them, there is a need for research into learning spa- tial knowledge representations from data, and for connecting them to language. Representing un- stated facts is a challenging problem unaddressed by prior work and the focus of our contribution. This problem is a counterpart to the image descrip- tion problem (Kulkarni et …

Content truncated to 500 words