Communication breakdown: On the low mutual intelligibility between human and neural captioning

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7998 - 8007 December 7-11, 2022 ©2022 Association for Computational Linguistics Communication breakdown: On the low mutual intelligibility between human and neural captioning Roberto Dessì Meta AI Universitat Pompeu Fabra rdessi@meta.com Eleonora Gualdoni and Francesca Franzon Universitat Pompeu Fabra {name.lastname}@upf.edu Gemma Boleda and Marco Baroni ICREA Universitat Pompeu Fabra {name.lastname}@upf.edu Abstract We compare the 0-shot performance of a neu- ral caption-based image retriever when given as input either human-produced captions or captions generated by a neural captioner. We conduct this comparison on the recently in- troduced IMAGECODE data-set (Krojer et al., 2022), which contains hard distractors nearly identical to the images to be retrieved. We find that the neural retriever has much higher perfor- mance when fed neural rather than human cap- tions, despite the fact that the former, unlike the latter, were generated without awareness of the distractors that make the task hard. Even more remarkably, when the same neural captions are given to human subjects, their retrieval perfor- mance is almost at chance level. Our results thus add to the growing body of evidence that, even when the “language” of neural models re- sembles English, this superficial resemblance might be deeply misleading. 1 Introduction Neural vision-and-language models have achieved impressive results in tasks such as visual common- sense reasoning and question answering (e.g., Chen et al., 2019; Lu et al., 2019). However, Krojer et al. (2022) recently showed, in the context of caption- based image retrieval, that state-of-the-art multi- modal models still perform poorly when the candi- date pool contains very similar distractor images (such as close frames from the same video). Here, we show that, when the best pre-trained image retrieval system of Krojer et al. (2022) is fed captions produced by an out-of-the box neural cap- tion generator, its performance makes a big jump forward. 0-shot image retrieval accuracy improves by almost 6% compared to the highest previously reported human-caption-based performance by the same model, with fine-tuning and various ad-hoc architectural adaptations. This is remarkable, be- cause the off-the-shelf caption generator we use (unlike the humans who wrote the original captions in the data-set) is not taking the set of distractor im- ages into account. Even more remarkably, we show that, when human subjects are tasked with retriev- ing the right image using the same neural captions that help the model so much, their performance is only marginally above chance level. 2 Setup Data We use the more challenging video sec- tion of the IMAGECODE data-set (Krojer et al., 2022). Since we do not fine-tune our model, we only use the validation set, including 1,872 data points.1 Henceforth, when when we employ the term IMAGECODE, we are referring to this subset. Each data-point consists of a target image and 9 distractors, where the target and the distractors are frames from the same (automatically segmented) scene in a video. We also use the human captions in the data-set, that were produced by subjects that …

Content truncated to 500 words