Cross-Modal Similarity-Based Curriculum Learning for Image Captioning

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7599 - 7606 December 7-11, 2022 ©2022 Association for Computational Linguistics Cross-Modal Similarity-Based Curriculum Learning for Image Captioning Hongkuan Zhang1 Saku Sugawara2 Akiko Aizawa2 Lei Zhou1 Ryohei Sasano1 Koichi Takeda1 1Nagoya University 2National Institute of Informatics {zhang.hongkuan.k5,zhou.lei.e1}@s.mail.nagoya-u.ac.jp {saku,aizawa}@nii.ac.jp {sasano,takedasu}@i.nagoya-u.ac.jp Abstract Image captioning models require the high-level generalization ability to describe the contents of various images in words. Most existing ap- proaches treat the image–caption pairs equally in their training without considering the dif- ferences in their learning difficulties. Several image captioning approaches introduce curricu- lum learning methods that present training data with increasing levels of difficulty. However, their difficulty measurements are either based on domain-specific features or prior model training. In this paper, we propose a simple yet efficient difficulty measurement for image cap- tioning using cross-modal similarity calculated by a pretrained vision–language model. Ex- periments on the COCO and Flickr30k datasets show that our proposed approach achieves supe- rior performance and competitive convergence speed to baselines without requiring heuristics or incurring additional training costs. More- over, the higher model performance on difficult examples and unseen data also demonstrates the generalization ability. 1 Introduction Image captioning has been widely investigated in computer vision and language research. However, most current methods treat image–caption pairs for training indistinctively, thus neglecting the differ- ence in terms of learning difficulty. As illustrated in Figure 1, an image is annotated with multiple references with diverse styles and complexity lev- els. Such diversity can introduce different levels of learning difficulty, and undertrained captioning models can be misled by wrong gradients when training on the difficult data (Dong et al., 2021). Curriculum learning (CL) has demonstrated im- provements in model performance and training speed by presenting data sorted according to the learning difficulty (Bengio et al., 2009). Existing image captioning approaches using CL have draw- backs in their difficulty measurements: 1) Requir- References: - A pizza with multiple toppings including an egg. - A white plate holding a pizza next to plate of fries. - A pizza sits on top of a pan covered in vegetables. - A pizza pie with vegetables of some sort on it. - A full veggie pizza is ready to be eaten. Pretrained Vision-Language Model Similarity: 32.5 28.2 28.0 26.0 25.5 References: - A city street lined with pine trees on the medium. - A large metal ball sitting next to a street in a major city. - A truck some palm trees buildings and traffic lights. - A truck driving down a busy street. - The image is not loading to describe it. Similarity: 26.0 22.4 17.9 16.2 14.0 Figure 1: Example of cross-modal similarity score for the caption data calculated by the pretrained VL model CLIP. Numbers with RED and BLUE colors denote the higher and lower scores respectively. ing domain-specific knowledge or heuristics (Liu et al., 2021); 2) Adding up the mono-modal diffi- culty scores without considering the cross-modal features (Alsharid et al., 2021); and 3) …

Content truncated to 500 words