Removing Word-Level Spurious Alignment between Images and Pseudo-Captions in Unsupervised Image Captioning

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 3692–3702 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 3692 Removing Word-Level Spurious Alignment between Images and Pseudo-Captions in Unsupervised Image Captioning Ukyo Honda1,3 Yoshitaka Ushiku2 Atsushi Hashimoto2 Taro Watanabe1 Yuji Matsumoto3 1 Nara Institute of Science and Technology 2 OMRON SINIC X Corp. 3 RIKEN Center for Advanced Intelligence Project 1 {honda.ukyo.hn6, taro}@is.naist.jp 2 {yoshitaka.ushiku, atsushi.hashimoto}@sinicx.com 3 yuji.matsumoto@riken.jp Abstract Unsupervised image captioning is a challeng- ing task that aims at generating captions with- out the supervision of image–sentence pairs, but only with images and sentences drawn from different sources and object labels de- tected from the images. In previous work, pseudo-captions, i.e., sentences that contain the detected object labels, were assigned to a given image. The focus of the previous work was on the alignment of input images and pseudo-captions at the sentence level. How- ever, pseudo-captions contain many words that are irrelevant to a given image. In this work, we investigate the effect of removing mis- matched words from image–sentence align- ment to determine how they make this task dif- ficult. We propose a simple gating mechanism that is trained to align image features with only the most reliable words in pseudo-captions: the detected object labels. The experimental results show that our proposed method out- performs the previous methods without intro- ducing complex sentence-level learning objec- tives. Combined with the sentence-level align- ment method of previous work, our method further improves its performance. These re- sults confirm the importance of careful align- ment in word-level details.1 1 Introduction Image captioning is a task to describe images in natural languages. This is a fundamental challenge with regard to automatically retrieving and summa- rizing the visual information in a human-readable form. Recently, considerable progress has been made (Vinyals et al., 2015; Xu et al., 2015; An- derson et al., 2018b) owing to the development of neural networks and a large number of annotated 1Code will be available at https://github.com/ ukyh/RemovingSpuriousAlignment image–sentence pairs (Young et al., 2014; Lin et al., 2014; Krishna et al., 2017). However, these pairs are limited in their coverage of scenes2, and scal- ing them is difficult owing to the cost of manual annotation. Unsupervised image captioning (Feng et al., 2019) aims to describe scenes that have no cor- responding image–sentence pairs, without requir- ing additional annotation of the pairs. The only available resources are images and sentences drawn from different sources and object labels detected from the images. Although it is highly challenging, unsupervised image captioning has the potential to cover a broad range of scenes by exploiting a large number of images and sentences that are not paired by expensive manual annotation. To train a captioning model in this setting, pre- vious work (Feng et al., 2019; Laina et al., 2019) employed sentences that contained the object la- bels detected from given images. We refer to these sentences as pseudo-captions. However, pseudo- captions are problematic in that …

Content truncated to 500 words