Annotation Tools and Knowledge Representation for a Text-To-Scene System

Proceedings of COLING 2012: Technical Papers, pages 679–694, COLING 2012, Mumbai, December 2012. Annotation Tools and Knowledge Representation for a Text-To-Scene System Bob Coyne1 Alex Klapheke1 Masoud Rouhizadeh2 Richard Sproat3 Daniel Bauer1 (1) Columbia University (2) Oregon Health Sciences University (3) Google coyne@cs.columbia.edu, agk2118@columbia.edu, masoud@cslu.ogi.edu, rws@xoba.com, bauer@cs.columbia.edu ABSTRACT Text-to-scene conversion requires knowledge about how actions and locations are expressed in language and realized in the world. To provide this knowlege, we are creating a lexical resource (VigNet) that extends FrameNet by creating a set of intermediate frames (vignettes) that bridge between the high-level semantics of FrameNet frames and a new set of low-level primitive graphical frames. Vignettes can be thought of as a link between function and form – between what a scene means and what it looks like. In this paper, we describe the set of primitive graphical frames and the functional properties of 3D objects (affordances) we use in this decomposition. We examine the methods and tools we have developed to populate VigNet with a large number of action and location vignettes. KEYWORDS: text-to-scene conversion, world knowledge, frame semantics, visual semantics, linguistic annotation, crowdsourcing. 679 Figure 1: Mocked-up scenes using the WASH-FRUIT-IN-SINK vignette (“John washes the apple”) and WASH-FLOOR-W-SPONGE vignette (“John washes the floor”). 1 Introduction 3D graphics authoring is a difficult process, requiring users to master a series of complex menus, dialog boxes, and often tedious direct manipulation techniques. Natural language offers an interface that is intuitive and immediately accessible to anyone, without requiring any special skill or training. The WordsEye system (Coyne and Sproat, 2001) lets users create 3D scenes by describing them in language. It has been used by several thousand users to create over 10,000 scenes by merely describing them. We have tested WordsEye as an educational tool with rising 5th grade children in a summer enrichment program where it was found to significantly improve literacy skills over the students who had taken the more traditional version of the course (Coyne et al., 2011b). As one of the students said “When you read a book, you don’t get any pictures. WordsEye helps you create your own pictures, so you can picture in your mind what happens in the story.” The students were also introduced to WordsEye’s face and emotion manipulation capabilities – the children loved including themselves and other people in scenes and modifying the facial expressions. We are currently experimenting with automatically depicting Twitter tweets as a way to bring text-to-scene visualization to a wider audience and to test the limits of the system in an open domain with ill-formed text. In this paper we describe a new set annotation tools and the graphical primitives we have developed in order to build a knowledge base that maps linguistic constructs into semantic frames representing spatial relations and other graphical relations. WordsEye currently focuses on directly expressed spatial relations and other graphically real- izable properties. As a result, users must describe scenes in somewhat stilted language. Our goal is to build a comprehensive text-to-scene system …

Content truncated to 500 words