Domain-Specific Image Captioning

Proceedings of the Eighteenth Conference on Computational Language Learning, pages 11–20, Baltimore, Maryland USA, June 26-27 2014. c⃝2014 Association for Computational Linguistics Domain-Specific Image Captioning Rebecca Mason and Eugene Charniak Brown Laboratory for Linguistic Information Processing (BLLIP) Brown University, Providence, RI 02912 {rebecca,ec}@cs.brown.edu Abstract We present a data-driven framework for image caption generation which incorpo- rates visual and textual features with vary- ing degrees of spatial structure. We pro- pose the task of domain-specific image captioning, where many relevant visual details cannot be captured by off-the-shelf general-domain entity detectors. We ex- tract previously-written descriptions from a database and adapt them to new query images, using a joint visual and textual bag-of-words model to determine the cor- rectness of individual words. We imple- ment our model using a large, unlabeled dataset of women’s shoes images and nat- ural language descriptions (Berg et al., 2010). Using both automatic and human evaluations, we show that our caption- ing method effectively deletes inaccurate words from extracted captions while main- taining a high level of detail in the gener- ated output. 1 Introduction Broadly, the task of image captioning is: given a query image, generate a natural language descrip- tion of the image’s visual content. Both the im- age understanding and language generation com- ponents of this task are challenging open problems in their respective fields. A wide variety of ap- proaches have been proposed in the literature, for both the specific task of caption generation as well as related problems in understanding images and text. Typically, image understanding systems use su- pervised algorithms to detect visual entities and concepts in images. However, these typically re- quire accurate hand-labeled training data, which is not available in most specific domains. Ideally, 1. Extract existing human-authored caption according to similarity of coarse visual features. Query Image Nearest-Neighbor Nearest-neighbor caption: This sporty sneaker clog keeps foot cool and comfortable and fully supported. 2. Estimate correctness of extracted words using domain- specific joint model of text and visual bag-of-word features. This sporty sneaker clog keeps foot cool and comfortable and fully supported. 3. Compress extracted caption to adapt its content while maintaining grammatical correctness. Output: This clog keeps foot comfortable and supported. a domain-specific image captioning system would learn in a less supervised fashion, using captioned images found on the web. This paper focuses on image caption genera- tion for a specific domain – images of women’s shoes, collected from online shopping websites. Our framework has three main components. We extract an existing description from a database of human-captions, by projecting query images into a multi-dimensional space where structurally similar images are near each other. We also train a joint topic model to discover the latent topics which generate both captions and images. We combine these two approaches using sentence compression to delete modifying details in the ex- tracted caption which are not relevant to the query image. Our captioning framework is inspired by sev- eral recent approaches at the intersection of Nat- ural Language Processing and Computer Vision. …

Content truncated to 500 words