Evaluating the WordsEye Text-to-Scene System: Imaginative and Realistic Sentences

Evaluating the WordsEye Text-to-Scene System: Imaginative and Realistic Sentences Morgan Ulinski, Bob Coyne, Julia Hirschberg Department of Computer Science Columbia University New York, NY, USA {mulinski, coyne, julia}@cs.columbia.edu Abstract We describe our evaluation of the WordsEye text-to-scene generation system. We address the problem of evaluating the output of such a system vs. simple search methods to find a picture to illustrate a sentence. To do this, we constructed two sets of test sentences: a set of crowdsourced imaginative sentences and a set of realistic sentences extracted from the PASCAL image caption corpus (Rashtchian et al., 2010). For each sentence, we compared sample pictures found using Google Image Search to those produced by WordsEye. We then crowdsourced judgments as to which picture best illustrated each sentence. For imaginative sentences, pictures produced by WordsEye were preferred, but for realistic sentences, Google Image Search results were preferred. We also used crowdsourc- ing to obtain a rating for how well each picture illustrated the sentence, from 1 (completely correct) to 5 (completely incorrect). WordsEye pictures had an average rating of 2.58 on imaginative sentences and 2.54 on realistic sentences; Google images had an aver- age rating of 3.82 on imaginative sentences and 1.87 on realistic sentences. We also discuss the sources of errors in the WordsEye system. Keywords: text-to-scene, evaluation, corpus creation, crowdsourcing 1. Introduction WordsEye (Coyne and Sproat, 2001) is a system for auto- matically converting natural language text into 3D scenes representing the meaning of that text. WordsEye supports language-based control of spatial relations, spatial proper- ties, surface textures and colors, and cardinality; it handles simple anaphora and coreference resolution, allowing for a variety of ways to refer to objects and describe scenes. Scenes are assembled from a library of 3,000 3D objects and 10,000 2D images tied to a lexicon of 15,000 nouns. WordsEye is a web application (http://www.wordseye.com) with 27,000 registered real-world users. During the two year period from November 2015 to December 2017, ap- proximately 2,200 users posted 14,000 finished scenes to an online gallery. One task that WordsEye addresses is the problem of cre- ating vs. automatically finding a picture to illustrate a sen- tence. Standard image search engines are limited to pic- tures that already exist in their databases, biasing them to- ward retrieving images of mundane and real-world scenar- ios. In contrast, a scene generation system like WordsEye can illustrate a much wider range of images, allowing users to visualize unusual and fantastical scenes. When users are freed from the normal constraints of what is possible or already exists they will often describe what they imag- ine – from situational, to iconic, to abstract, to fantastical. The majority of scenes created by actual users of the online WordsEye system are imaginative. One user commented “I truly enjoy watching people unleash their minds here.” Some examples of imaginative scenes that have been cre- ated in WordsEye are shown in Figure 1. The ability to generate both realistic and imaginative scenes from text input demonstrates that text-to-scene generation …

Content truncated to 500 words