FOIL it! Find One mismatch between Image and Language caption

Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 255–265 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1024 Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 255–265 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1024 FOIL it! Find One mismatch between Image and Language caption Ravi Shekhar, Sandro Pezzelle, Yauhen Klimovich, Aur´elie Herbelot, Moin Nabi, Enver Sangineto, Raffaella Bernardi University of Trento {firstname.lastname}@unitn.it Abstract In this paper, we aim to understand whether current language and vision (LaVi) models truly grasp the interac- tion between the two modalities. To this end, we propose an extension of the MS- COCO dataset, FOIL-COCO, which asso- ciates images with both correct and ‘foil’ captions, that is, descriptions of the im- age that are highly similar to the original ones, but contain one single mistake (‘foil word’). We show that current LaVi mod- els fall into the traps of this data and per- form badly on three tasks: a) caption clas- sification (correct vs. foil); b) foil word detection; c) foil word correction. Hu- mans, in contrast, have near-perfect per- formance on those tasks. We demonstrate that merely utilising language cues is not enough to model FOIL-COCO and that it challenges the state-of-the-art by requiring a fine-grained understanding of the rela- tion between text and image. 1 Introduction Most human language understanding is grounded in perception. There is thus growing interest in combining information from language and vision in the NLP and AI communities. So far, the primary testbeds of Language and Vision (LaVi) models have been ‘Visual Question Answering’ (VQA) (e.g. Antol et al. (2015); Mali- nowski and Fritz (2014); Malinowski et al. (2015); Gao et al. (2015); Ren et al. (2015)) and ‘Im- age Captioning’ (IC) (e.g. Hodosh et al. (2013); Fang et al. (2015); Chen and Lawrence Zitnick (2015); Donahue et al. (2015); Karpathy and Fei-Fei (2015); Vinyals et al. (2015)). Whilst some models have seemed extremely successful on those tasks, it remains unclear how the re- ported results should be interpreted and what those Figure 1: Is the caption correct or foil (T1)? If it is foil, where is the mistake (T2) and which is the word to correct the foil one (T3)? models are actually learning. There is an emerg- ing feeling in the community that the VQA task should be revisited, especially as many current dataset can be handled by ‘blind’ models which use language input only, or by simple concate- nation of language and vision features (Agrawal et al., 2016; Jabri et al., 2016; Zhang et al., 2016; Goyal et al., 2016a). In IC too, Hodosh and Hock- enmaier (2016) showed that, contrarily to what prior research had suggested, the task is far from been solved, since IC models are not able to dis- tinguish between a correct and incorrect caption. Such results indicate that in current datasets, language provides priors that make LaVi models successful without …

Content truncated to 500 words