Learning to Relate Literal and Sentimental Descriptions of Visual Properties

Proceedings of NAACL-HLT 2013, pages 416–425, Atlanta, Georgia, 9–14 June 2013. c⃝2013 Association for Computational Linguistics Learning to Relate Literal and Sentimental Descriptions of Visual Properties Mark Yatskar Computer Science & Engineering University of Washington Seattle, WA my89@cs.washington.edu Svitlana Volkova Center for Language and Speech Processing Johns Hopkins University Baltimore, MD svitlana@jhu.edu Asli Celikyilmaz Conversational Understanding Sciences Microsoft Mountain View, CA asli@ieee.org Bill Dolan NLP Group Microsoft Research Redmond, WA billdol@microsoft.edu Luke Zettlemoyer Computer Science & Engineering University of Washington Seattle, WA lsz@cs.washington.edu Abstract Language can describe our visual world at many levels, including not only what is lit- erally there but also the sentiment that it in- vokes. In this paper, we study visual language, both literal and sentimental, that describes the overall appearance and style of virtual char- acters. Sentimental properties, including la- bels such as “youthful” or “country western,” must be inferred from descriptions of the more literal properties, such as facial features and clothing selection. We present a new dataset, collected to describe Xbox avatars, as well as models for learning the relationships between these avatars and their literal and sentimen- tal descriptions. In a series of experiments, we demonstrate that such learned models can be used for a range of tasks, including pre- dicting sentimental words and using them to rank and build avatars. Together, these re- sults demonstrate that sentimental language provides a concise (though noisy) means of specifying low-level visual properties. 1 Introduction Language can describe varied aspects of our visual world, including not only what is literally there but also the social, cultural, and emotional sentiment it invokes. Recently, there has been a growing effort to study literal language that describes directly ob- servable properties, such as object color, shape, or This is a light tan young man with short and trim haircut. He has straight eyebrows and large brown eyes. He has a neat and trim appearance. State of mind: angry, upset, determined. Likes: country western, rodeo. Occupation: cowboy, wrangler, horse trainer. Overall: youthful, cowboy. Figure 1: (A) Literal avatar descriptions and (B) sen- timental descriptions of four avatar properties, in- cluding possible occupations and interests. category (Farhadi et al., 2009; Mitchell et al., 2010; Matuszek et al., 2012). Here, we add a focus on sentimental visual language, which compactly de- scribes more subjective properties such as if a person looks determined, if a resume looks professional, or if a restaurant looks romantic. Such models enable many new applications, such as text editors that au- tomatically select properties including font, color, or text alignment to best match high level descriptions such as “professional” or “artistic.” 416 In this paper, we study visual language, both lit- eral and sentimental, that describes the overall ap- pearance and style of virtual characters, like those in Figure 1. We use literal language as feature norms, a tool used for studying semantic information in cog- nitive science (Mcrae et al., 2005). Literal words, such “black” or “hat,” are annotated for objects to in- dicate how people …

Content truncated to 500 words