TreeTalk: Composition and Compression of Trees for Image Descriptions

TREETALK: Composition and Compression of Trees for Image Descriptions Polina Kuznetsova† † Stony Brook University Stony Brook, NY pkuznetsova @cs.stonybrook.edu Vicente Ordonez‡ Tamara L. Berg‡ ‡ UNC Chapel Hill Chapel Hill, NC {vicente,tlberg} @cs.unc.edu Yejin Choi†† ††University of Washington Seattle, WA yejin@cs.washington.edu Abstract We present a new tree based approach to composing expressive image descriptions that makes use of naturally occuring web images with captions. We investigate two related tasks: image caption generalization and gen- eration, where the former is an optional sub- task of the latter. The high-level idea of our approach is to harvest expressive phrases (as tree fragments) from existing image descrip- tions, then to compose a new description by selectively combining the extracted (and op- tionally pruned) tree fragments. Key algo- rithmic components are tree composition and compression, both integrating tree structure with sequence structure. Our proposed system attains significantly better performance than previous approaches for both image caption generalization and generation. In addition, our work is the first to show the empirical ben- efit of automatically generalized captions for composing natural image descriptions. 1 Introduction The web is increasingly visual, with hundreds of bil- lions of user contributed photographs hosted online. A substantial portion of these images have some sort of accompanying text, ranging from keywords, to free text on web pages, to textual descriptions di- rectly describing depicted image content (i.e. cap- tions). We tap into the last kind of text, using natu- rally occuring pairs of images with natural language descriptions to compose expressive descriptions for query images via tree composition and compression. Such automatic image captioning efforts could potentially be useful for many applications: from automatic organization of photo collections, to facil- itating image search with complex natural language queries, to enhancing web accessibility for the vi- sually impaired. On the intellectual side, by learn- ing to describe the visual world from naturally exist- ing web data, our study extends the domains of lan- guage grounding to the highly expressive language that people use in their everyday online activities. There has been a recent spike in efforts to au- tomatically describe visual content in natural lan- guage (Yang et al., 2011; Kulkarni et al., 2011; Li et al., 2011; Farhadi et al., 2010; Krishnamoorthy et al., 2013; Elliott and Keller, 2013; Yu and Siskind, 2013; Socher et al., 2014). This reflects the long standing understanding that encoding the complex- ities and subtleties of image content often requires more expressive language constructs than a set of tags. Now that visual recognition algorithms are be- ginning to produce reliable estimates of image con- tent (Perronnin et al., 2012; Deng et al., 2012a; Deng et al., 2010; Krizhevsky et al., 2012), the time seems ripe to begin exploring higher level semantic tasks. There have been two main complementary direc- tions explored for automatic image captioning. The first focuses on describing exactly those items (e.g., objects, attributes) that are detected by vision recog- nition, which subsequently confines what should be described and how (Yao et al., …

Content truncated to 500 words