Decoupled Box Proposal and Featurization with Ultrafine-Grained Semantic Labels Improve Image Captioning and Visual Question Answering

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 1468–1474, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 1468 Decoupled Box Proposal and Featurization with Ultrafine-Grained Semantic Labels Improve Image Captioning and Visual Question Answering Soravit Changpinyo Bo Pang Piyush Sharma Radu Soricut Google AI Venice, CA 90291 {schangpi,bopang,piyushsharma,rsoricut}@google.com Abstract Object detection plays an important role in current solutions to vision and language tasks like image captioning and visual question an- swering. However, popular models like Faster R-CNN rely on a costly process of annotat- ing ground-truths for both the bounding boxes and their corresponding semantic labels, mak- ing it less amenable as a primitive task for transfer learning. In this paper, we examine the effect of decoupling box proposal and fea- turization for down-stream tasks. The key in- sight is that this allows us to leverage a large amount of labeled annotations that were previ- ously unavailable for standard object detection benchmarks. Empirically, we demonstrate that this leads to effective transfer learning and im- proved image captioning and visual question answering models, as measured on publicly- available benchmarks. 1 Introduction Object detection has been employed extensively as a primitive task for vision and language tasks such as image captioning and visual question answering (VQA); see (Anderson et al., 2018) and the work that follows it. One motivation is that the abil- ity to recognize salient regions and objects may be too difficult to learn from weakly-supervised top- down signals, in the form of captions and question- answer pairs. Indeed, bottom-up signals provided by object detection often correspond to semantic units of language such as words or phrases, mak- ing them suitable for text generation and image- text alignment. However, object detection itself can be broken down into multiple subtasks (Liu et al., 2018). A family of “two-stage” object detectors first pro- poses category-agnostic bounding box candidates and then featurizes and classifies the cropped re- gions into one of the available semantic labels. Q: How much money is this? Q: What is this? FRCNN (VG) A: 1 dollar A: beer Ultra A: 20 A: bbq sauce Figure 1: Ultrafine-grained semantic labels (at “in- stance level”) provide transfer learning power to down- stream tasks like visual question answering. Even “one-stage” object detection approaches, where these boxes become category-specific, can be formulated in a bottom-up manner as detect- ing and grouping extreme and center points (Zhou et al., 2019). Can we take advantage of this obser- vation to learn to transfer more effectively? In this work, we take a step in this direction by examining the effect of decoupling box proposal and featur- ization on downstream vision and language tasks. In particular, we consider a two-stage object de- tector and set a goal of pushing the “featurization” aspect of the task further than before. Our choice to break free from “featurization by object detection models” has at least two advan- tages. First, there is a …

Content truncated to 500 words