Obj2Text: Generating Visually Descriptive Language from Object Layouts

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 177–187 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics OBJ2TEXT: Generating Visually Descriptive Language from Object Layouts Xuwang Yin Vicente Ordonez Department of Computer Science, University of Virginia, Charlottesville, VA. [xy4cm, vicente]@virginia.edu Abstract Generating captions for images is a task that has recently received considerable at- tention. In this work we focus on caption generation for abstract scenes, or object layouts where the only information pro- vided is a set of objects and their locations. We propose OBJ2TEXT, a sequence-to- sequence model that encodes a set of ob- jects and their locations as an input se- quence using an LSTM network, and de- codes this representation using an LSTM language model. We show that our model, despite encoding object layouts as a se- quence, can represent spatial relationships between objects, and generate descriptions that are globally coherent and semanti- cally relevant. We test our approach in a task of object-layout captioning by us- ing only object annotations as inputs. We additionally show that our model, com- bined with a state-of-the-art object detec- tor, improves an image captioning model from 0.863 to 0.950 (CIDEr score) in the test benchmark of the standard MS-COCO Captioning task. 1 Introduction Natural Language generation (NLG) is a long standing goal in natural language processing. There have already been several successes in ap- plications such as financial reporting (Kukich, 1983; Smadja and McKeown, 1990), or weather forecasts (Konstas and Lapata, 2012; Wen et al., 2015), however it is still a challenging task for less structured and open domains. Given recent progress in training robust visual recognition mod- els using convolutional neural networks, the task of generating natural language descriptions for ar- Word Embedding Location Encoder group of people are flying a kite END a Encoder Output START person person kite (a) Input Object Layout (b) LSTM Object Layout Encoder (c) LSTM Language Model Decoder Figure 1: Overview of our proposed model for generating visually descriptive language from ob- ject layouts. The input (a) is an object layout that consists of object categories and their correspond- ing bounding boxes, the encoder (b) uses a two- stream recurrent neural network to encode the in- put object layout, and the decoder (c) uses a stan- dard LSTM recurrent neural network to generate text. bitrary images has received considerable atten- tion (Vinyals et al., 2015; Karpathy and Fei-Fei, 2015; Mao et al., 2015). In general, generating vi- sually descriptive language can be useful for vari- ous tasks such as human-machine communication, accessibility, image retrieval, and search. How- ever this task is still challenging and it depends on developing both a robust visual recognition model, and a reliable language generation model. In this paper, we instead tackle a task of describing ob- ject layouts where the categories for the objects in an input scene and their corresponding locations are known. Object layouts are commonly used for story-boarding, sketching, and computer graphics applications. Additionally, using our object …

Content truncated to 500 words