Text to 3D Scene Generation with Rich Lexical Grounding

Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 53–62, Beijing, China, July 26-31, 2015. c⃝2015 Association for Computational Linguistics Text to 3D Scene Generation with Rich Lexical Grounding Angel Chang∗, Will Monroe∗, Manolis Savva, Christopher Potts and Christopher D. Manning Stanford University, Stanford, CA 94305 {angelx,wmonroe4,msavva}@cs.stanford.edu, {cgpotts,manning}@stanford.edu Abstract The ability to map descriptions of scenes to 3D geometric representations has many applications in areas such as art, educa- tion, and robotics. However, prior work on the text to 3D scene generation task has used manually specified object cate- gories and language that identifies them. We introduce a dataset of 3D scenes an- notated with natural language descriptions and learn from this data how to ground tex- tual descriptions to physical objects. Our method successfully grounds a variety of lexical terms to concrete referents, and we show quantitatively that our method im- proves 3D scene generation over previ- ous work using purely rule-based meth- ods. We evaluate the fidelity and plau- sibility of 3D scenes generated with our grounding approach through human judg- ments. To ease evaluation on this task, we also introduce an automated metric that strongly correlates with human judgments. 1 Introduction We examine the task of text to 3D scene gener- ation. The ability to map descriptions of scenes to 3D geometric representations has a wide vari- ety of applications; many creative industries use 3D scenes. Robotics applications need to interpret commands referring to real-world environments, and the ability to visualize scenarios given high- level descriptions is of great practical use in educa- tional tools. Unfortunately, 3D scene design user interfaces are prohibitively complex for novice users. Prior work has shown the task remains chal- lenging and time intensive for non-experts, even with simplified interfaces (Savva et al., 2014). ∗The first two authors are listed in alphabetical order. {...L-shaped room with walls that have 2 tones of gray..., A dark room with a pool table...} {...a multicolored table in the middle of the room , ...four red and white chairs and a colorful table, ...} Figure 1: We learn how to ground references such as “L-shaped room” to 3D models in a paired cor- pus of 3D scenes and natural language descrip- tions. Sentence fragments in bold were identified as high-weighted references to the shown objects. Language offers a convenient way for designers to express their creative goals. Systems that can interpret natural descriptions to build a visual rep- resentation allow non-experts to visually express their thoughts with language, as was demonstrated by WordsEye, a pioneering work in text to 3D scene generation (Coyne and Sproat, 2001). WordsEye and other prior work in this area (Seversky and Yin, 2006; Chang et al., 2014) used manually chosen mappings between lan- guage and objects in scenes. To our knowledge, we present the first 3D scene generation approach that learns from data how to map textual terms to objects. First, we collect a dataset of 3D …

Content truncated to 500 words