Image Captioning with Very Scarce Supervised Data: Adversarial Semi-Supervised Learning Approach

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 2012–2023, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 2012 Image Captioning with Very Scarce Supervised Data: Adversarial Semi-Supervised Learning Approach Dong-Jin Kim1 Jinsoo Choi1 Tae-Hyun Oh2 In So Kweon1 1KAIST, South Korea. 2MIT CSAIL, Cambridge, MA. 1{djnjusa,jinsc37,iskweon77}@kaist.ac.kr 2taehyun@csail.mit.edu Abstract Constructing an organized dataset comprised of a large number of images and several cap- tions for each image is a laborious task, which requires vast human effort. On the other hand, collecting a large number of images and sentences separately may be immensely eas- ier. In this paper, we develop a novel data- efficient semi-supervised framework for train- ing an image captioning model. We leverage massive unpaired image and caption data by learning to associate them. To this end, our proposed semi-supervised learning method as- signs pseudo-labels to unpaired samples via Generative Adversarial Networks to learn the joint distribution of image and caption. To evaluate, we construct scarcely-paired COCO dataset, a modified version of MS COCO cap- tion dataset. The empirical results show the ef- fectiveness of our method compared to several strong baselines, especially when the amount of the paired samples are scarce. 1 Introduction Image captioning is a task of automatically gener- ating a natural language description of a given im- age. It is a highly useful task, in that 1) it extracts the essence from an image into a self-descriptive form of representation, and 2) the output format is a natural language, which exhibits free-form and manageable characteristics useful to applica- tions such as language based image or region re- trieval [Johnson et al., 2016; Karpathy and Fei- Fei, 2015; Kim et al., 2019], video summariza- tion [Choi et al., 2018], navigation [Wang et al., 2018a], vehicle control [Kim et al., 2018b]. Im- age captioning exhibits free-form characteristics, since it is not confined to a few number of pre- defined classes. This enables descriptive analysis of a given image. Recent research on image captioning has made impressive progress [Anderson et al., 2018; Vinyals et al., 2015]. Despite this progress, the Paired Data (𝑫𝒑) Unpaired Image Data (𝑫𝒖𝒙) Unpaired Text Data (𝑫𝒖 𝒙���) A pizza with basil leaves served on a plate. A woman cutting a large cake with one lit candle. A piece of cake and coffee are on an outdoor table. A white dog has a purple Frisbee in it’s mouth. A chocolate cake and a fork ready to be eat. A woman cutting a pizza with a fork and knife. Figure 1: The proposed data setup utilizes both “paired” and “unpaired” image-caption data. We de- note paired data as Dp, and unpaired image and caption datasets as Dx u and Dy u respectively. majority of works are trained via supervised learn- ing that requires a large corpus of caption labeled images such as the MS COCO caption dataset [Lin et al., 2014]. Specifically, the dataset is con- structed with …

Content truncated to 500 words