CapWAP: Image Captioning with a Purpose

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 8755–8768, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 8755 CAPWAP: Captioning with a Purpose Adam Fisch1∗ Kenton Lee2 Ming-Wei Chang2 Jonathan H. Clark2 Regina Barzilay1 1Massachusetts Institute of Technology, 2Google Research {fisch, regina}@csail.mit.edu, {kentonl, mingweichang, jhclark}@google.com Abstract The traditional image captioning task uses generic reference captions to provide textual information about images. Different user pop- ulations, however, will care about different visual aspects of images. In this paper, we propose a new task, Captioning with A Pur- pose (CAPWAP). Our goal is to develop sys- tems that can be tailored to be useful for the information needs of an intended population, rather than merely provide generic information about an image. In this task, we use question- answer (QA) pairs—a natural expression of information need—from users, instead of ref- erence captions, for both training and post- inference evaluation. We show that it is pos- sible to use reinforcement learning to directly optimize for the intended information need, by rewarding outputs that allow a question an- swering model to provide correct answers to sampled user questions. We convert several visual question answering datasets into CAP- WAP datasets, and demonstrate that under a variety of scenarios our purposeful captioning system learns to anticipate and fulfill specific information needs better than its generic coun- terparts, as measured by QA performance on user questions from unseen images, when us- ing the caption alone as context. 1 Introduction The image captioning task typically selects for cap- tions having high similarity with generic human references. While this task definition has driven much of the research in the field, the end-purpose of these captions is not always clearly articulated. We argue that (1) generic annotations may not be representative of users’ information needs, (2) user questions are a more natural way of articulating information needs, and (3) optimizing captions to provide correct answers to those questions allows ∗Work primarily completed while interning at Google. Task Caption Information Need Captioning There is a green bus. (Unspecified) Visual QA (Unspecified) Where’s it headed? CAPWAP At least three people are boarding the #14 bus to Bembridge. Which bus is this? Where’s it headed? How many people are boarding? Figure 1: The informational purpose of generic caption- ing is not clearly defined, and VQA provides only re- actionary information. The objective of the CAPWAP task is ultimately to provide more informative captions that specifically anticipate and satisfy users’ potential needs. In CAPWAP, we use QA as an implicit signal for information need: e.g., in the image above, a good caption that has been generated in advance should be able to be used to answer, Where is this bus headed? training to focus on information need. For example, in the VizWiz mobile application (Bigham et al., 2010), visually impaired users upload images from their everyday lives, along with questions about them that need to be answered. These questions serve as a powerful signal for aspects of the image that …

Content truncated to 500 words