Learning to Interpret and Describe Abstract Scenes

Human Language Technologies: The 2015 Annual Conference of the North American Chapter of the ACL, pages 1505–1515, Denver, Colorado, May 31 – June 5, 2015. c⃝2015 Association for Computational Linguistics Learning to Interpret and Describe Abstract Scenes Luis Gilberto Mateos Ortiz, Clemens Wolff and Mirella Lapata School of Informatics, University of Edinburgh 10 Crichton Street, Edinburgh EH8 9AB {clemens.wolff,luismattor}@gmail.com, mlap@inf.ed.ac.uk Abstract Given a (static) scene, a human can effort- lessly describe what is going on (who is do- ing what to whom, how, and why). The pro- cess requires knowledge about the world, how it is perceived, and described. In this paper we study the problem of interpreting and verbal- izing visual information using abstract scenes created from collections of clip art images. We propose a model inspired by machine trans- lation operating over a large parallel corpus of visual relations and linguistic descriptions. We demonstrate that this approach produces human-like scene descriptions which are both fluent and relevant, outperforming a num- ber of competitive alternatives based on tem- plates, sentence-based retrieval, and a multi- modal neural language model. 1 Introduction What is going on in the scene in Figure 1? Is the boy trying to feed the dog or play with it? Why is the girl upset? Is it because the dog is wearing her glasses? Or perhaps she is just scared of the dog? Scene interpretation is effortless for humans, almost everyone can summarize Figure 1 in a few words, without probably paying too much attention to the fact the girl is wearing a pink dress, the sun is yellow or that there is a plane in the sky. Discovering what an image means and relaying it in words is of theoretical importance raising ques- tions about language and its grounding in the per- ceptual world but also has practical applications. Examples include sentence-based image search and tools that enhance the accessibility of the web for visually impaired (blind and partially sighted) in- dividuals. Indeed, there has been a recent surge of interest in the development of models that au- tomatically describe image content in natural lan- Figure 1: Given an image, humans do not simply see an arrangement of objects, they understand how they relate to each other as well as their attributes and the activities they are involved in. guage (see references in Section 2). Due to the com- plex nature of the problem, existing approaches re- sort to modeling simplifications, on the generation side (e.g., through the use of templates and sentence- based retrieval methods), or the image processing side (e.g., by avoiding object-detection), or both. In this paper we study the problem of interpreting visual scenes and rendering their content using nat- ural language. We approach this problem within the methodology of Zitnick and Parikh (2013), who pro- posed the use of abstract scenes generated from clip art to model scene understanding (see Figure 1). The use of abstract scenes offers several advantages over real images. Firstly, it allows us to study the scene …

Content truncated to 500 words