End-to-end Image Captioning Exploits Distributional Similarity in Multimodal Space

Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 381–383 Brussels, Belgium, November 1, 2018. c⃝2018 Association for Computational Linguistics 381 End-to-end Image Captioning Exploits Distributional Similarity in Multimodal Space ∗ Pranava Madhyastha Josiah Wang Department of Computer Science University of Sheffield {p.madhyastha,j.k.wang,l.specia}@sheffield.ac.uk Lucia Specia Image description generation, or image caption- ing (IC), is the task of automatically generating a textual description for a given image. The gener- ated text is expected to describe, generally in a sin- gle sentence, what is visually depicted in the im- age, for example the entities/objects present in the image, their attributes, the actions/activities per- formed, entity/object interactions (including quan- tification), the location/scene, etc. (e.g. “a man riding a bike on the street”). Significant progress has been made with end-to-end approaches to tackling this problem, where parallel image– description datasets such as Flickr30k (Young et al., 2014) and MSCOCO (Chen et al., 2015) are used to train a CNN-RNN based neural network IC system (Vinyals et al., 2017; Karpathy and Fei-Fei, 2015; Xu et al., 2015). Such systems have demon- strated impressive performance in the COCO cap- tioning challenge1 according to automatic metrics, seemingly even surpassing human performance in many instances (e.g. CIDEr score > 1.0 vs. hu- man’s 0.85) (Chen et al., 2015). However, in real- ity, the performance of end-to-end systems is still far from satisfactory according to metrics based on human judgement2. This task is thus currently far from being a solved problem. We challenge the common assumption that end- to-end IC systems are able to achieve strong per- formance because they have learned to ‘under- stand’ and infer semantic information from visual representations, i.e. they can for example induce that “a boy is playing football” by learning di- rectly from mid-level image features and the corre- sponding textual descriptions in an implicit man- ner, without explicitly modeling the presence of ∗This is an abridged version of a recently published BMVC paper (Madhyastha et al., 2018) 1http://cocodataset.org/ #captions-challenge2015 2http://cocodataset.org/ #captions-leaderboard boy, ball, green field, etc. in the image. It is be- lieved that IC models have managed to infer that the phrase football is associated with some ‘green- like’ area in the image and is thus generated in the output description, or that the word boy is gen- erated because of some CNN activations corre- sponding to a young person. However, there seems to be no concrete evidence that this is the case. Instead, we hypothesize that the apparent strong performance of end-to-end systems is attributed to the fact that they exploit the distributional simi- larity in the multimodal feature space. To our best knowledge, our work is the first to provide empir- ical analysis of visual representations for the task of image captioning. By ‘distributional similarity’ we mean that IC models essentially attempt to match images from the training set that are most similar to a test im- age, and generate a caption from the most similar training instances, or generate a ‘novel’ descrip- …

Content truncated to 500 words