Object Counts! Bringing Explicit Detections Back into Image Captioning

Proceedings of NAACL-HLT 2018, pages 2180–2193 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Object Counts! Bringing Explicit Detections Back into Image Captioning Josiah Wang, Pranava Madhyastha and Lucia Specia Department of Computer Science University of Sheffield, UK {j.k.wang, p.madhyastha, l.specia}@sheffield.ac.uk Abstract The use of explicit object detectors as an in- termediate step to image captioning – which used to constitute an essential stage in early work – is often bypassed in the currently dom- inant end-to-end approaches, where the lan- guage model is conditioned directly on a mid- level image embedding. We argue that explicit detections provide rich semantic information, and can thus be used as an interpretable repre- sentation to better understand why end-to-end image captioning systems work well. We pro- vide an in-depth analysis of end-to-end image captioning by exploring a variety of cues that can be derived from such object detections. Our study reveals that end-to-end image cap- tioning systems rely on matching image rep- resentations to generate captions, and that en- coding the frequency, size and position of ob- jects are complementary and all play a role in forming a good image representation. It also reveals that different object categories con- tribute in different ways towards image cap- tioning. 1 Introduction Image captioning (IC), or image description gen- eration, is the task of automatically generating a sentential textual description for a given im- age. Early work on IC tackled the task by first running object detectors on the image and then using the resulting explicit detections as in- put to generate a novel textual description, e.g. (Kulkarni et al., 2011; Yang et al., 2011). With the advent of sequence-to-sequence approaches to IC, e.g. (Karpathy and Fei-Fei, 2015; Vinyals et al., 2015), coupled with the availability of large image description datasets, the performance of IC systems showed marked improvement, at least according to automatic evaluation metrics like Meteor (Denkowski and Lavie, 2014) and CIDEr (Vedantam et al., 2015). The currently dominant neural-based IC sys- tems are often trained end-to-end, using parallel (image, caption) datasets. Such systems are es- sentially sequential language models conditioned directly on some mid-level image features, such as an image embedding extracted from a pre-trained Convolutional Neural Network (CNN). Thus, they bypass the explicit detection phase of previous methods and instead generate captions directly from image features. Despite significant progress, it remains unclear why such systems work. major problem with these IC systems is that theyA are less interpretable than conventional pipelined methods which use explicit detections. We believe that it is timely to again start explor- ing the use of explicit object detections for image captioning. Explicit detections offer rich semantic information, which can be used to model the enti- ties in the image as well as their interactions, and can be used to better understand image captioning. Recent work (Yin and Ordonez, 2017) showed that conditioning an end-to-end IC model on vi- sual representations that implicitly encode object details yields reasonably good captions. Never- theless, it …

Content truncated to 500 words