Describing Images using Inferred Visual Dependency Representations

Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 42–52, Beijing, China, July 26-31, 2015. c⃝2015 Association for Computational Linguistics Describing Images using Inferred Visual Dependency Representations Desmond Elliott and Arjen P. de Vries Information Access Group Centrum Wiskunde & Informatica Amsterdam, The Netherlands elliott@cwi.nl, arjen@acm.org Abstract The Visual Dependency Representation (VDR) is an explicit model of the spa- tial relationships between objects in an im- age. In this paper we present an approach to training a VDR Parsing Model without the extensive human supervision used in previous work. Our approach is to find the objects mentioned in a given descrip- tion using a state-of-the-art object detec- tor, and to use successful detections to pro- duce training data. The description of an unseen image is produced by first predict- ing its VDR over automatically detected objects, and then generating the text with a template-based generation model using the predicted VDR. The performance of our approach is comparable to a state-of- the-art multimodal deep neural network in images depicting actions. 1 Introduction Humans typically write the text accompanying an image, which is a time-consuming and expen- sive activity. There are many circumstances in which people are well-suited to this task, such as captioning news articles (Feng and Lapata, 2008) where there are complex relationships between the modalities (Marsh and White, 2003). In this pa- per we focus on generating literal descriptions, which are rarely found alongside images because they describe what can easily be seen by others (Panofsky, 1939; Shatford, 1986; Hodosh et al., 2013). A computer that can automatically gen- erate these literal descriptions, filling the gap left by humans, may improve access to existing image collections or increase information access for vi- sually impaired users. There has been an upsurge of research in this area, including models that rely on spatial rela- tionships (Farhadi et al., 2010), corpus-based rela- tionships (Yang et al., 2011), spatial and visual at- tributes (Kulkarni et al., 2011), n-gram phrase fu- sion from Web-scale corpora (Li et al., 2011), tree- substitution grammars (Mitchell et al., 2012), se- lecting and combining phrases from large image- description collections (Kuznetsova et al., 2012), using Visual Dependency Representations to cap- ture spatial and corpus-based relationships (El- liott and Keller, 2013), and in a generative frame- work over densely-labelled data (Yatskar et al., 2014). The most recent developments have fo- cused on deep learning the relationships between visual feature vectors and word-embeddings with language generation models based on recurrent neural networks or long-short term memory net- works (Karpathy and Fei-Fei, 2015; Vinyals et al., 2015; Mao et al., 2015; Fang et al., 2015; Don- ahue et al., 2015; Lebret et al., 2015). An alter- native thread of research has focused on directly pairing images with text, based on kCCA (Hodosh et al., 2013) or multimodal deep neural networks (Socher et al., 2014; Karpathy et al., 2014). We revisit the Visual Dependency Representa- tion (Elliott and …

Content truncated to 500 words