Guided Open Vocabulary Image Captioning with Constrained Beam Search

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 936–945 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Guided Open Vocabulary Image Captioning with Constrained Beam Search Peter Anderson1, Basura Fernando1, Mark Johnson2, Stephen Gould1 1The Australian National University, Canberra, Australia firstname.lastname@anu.edu.au 2Macquarie University, Sydney, Australia mark.johnson@mq.edu.au Abstract Existing image captioning models do not generalize well to out-of-domain images containing novel scenes or objects. This limitation severely hinders the use of these models in real world applications deal- ing with images in the wild. We address this problem using a flexible approach that enables existing deep captioning ar- chitectures to take advantage of image tag- gers at test time, without re-training. Our method uses constrained beam search to force the inclusion of selected tag words in the output, and fixed, pretrained word embeddings to facilitate vocabulary ex- pansion to previously unseen tag words. Using this approach we achieve state of the art results for out-of-domain caption- ing on MSCOCO (and improved results for in-domain captioning). Perhaps sur- prisingly, our results significantly outper- form approaches that incorporate the same tag predictions into the learning algorithm. We also show that we can significantly im- prove the quality of generated ImageNet captions by leveraging ground-truth la- bels. 1 Introduction Automatic image captioning is a fundamental task that couples visual and linguistic learning. Re- cently, models incorporating recurrent neural net- works (RNNs) have demonstrated promising re- sults on this challenging task (Vinyals et al., 2015; Fang et al., 2015; Devlin et al., 2015), leverag- ing new benchmark datasets such as the MSCOCO dataset (Lin et al., 2014). However, these datasets are generally only concerned with a relatively small number of objects and interactions. Unsur- Input image containing previously unseen object (‘suitcase’) CNN-RNN Captioning Model A cat sitting inside of a suitcase. cat, suitcase, inside Constrained Beam Search Beam Search A cat sitting on top of a refrigerator. Image Tags Figure 1: We successfully caption images contain- ing previously unseen objects by incorporating se- mantic attributes (i.e., image tags) during RNN de- coding. Actual example from Section 4.2. prisingly, models trained on these datasets do not generalize well to out-of-domain images contain- ing novel scenes or objects (Tran et al., 2016). This limitation severely hinders the use of these models in real world applications dealing with im- ages in the wild. Although available image-caption training data is limited, many image collections are augmented with ground-truth text fragments such as semantic attributes (i.e., image tags) or object annotations. Even if these annotations do not exist, they can be generated using (potentially task specific) image taggers (Chen et al., 2013; Zhang et al., 2016) or object detectors (Ren et al., 2015; Krause et al., 2016), which are easier to scale to new concepts. In this paper our goal is to incorporate text frag- ments such as these during caption generation, to improve the quality of resulting captions. This goal poses two key challenges. First, RNNs are generally opaque, and difficult to …

Content truncated to 500 words