Improving Cross-modal Alignment for Text-Guided Image Inpainting

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 3445–3456 May 2-6, 2023 ©2023 Association for Computational Linguistics Improving Cross-modal Alignment for Text-Guided Image Inpainting Yucheng Zhou, Guodong Long∗ Australian AI Institute, School of Computer Science, FEIT, University of Technology Sydney yucheng.zhou-1@student.uts.edu.au, guodong.long@uts.edu.au Abstract Text-guided image inpainting (TGII) aims to restore missing regions based on a given text in a damaged image. Existing methods are based on a strong vision encoder and a cross- modal fusion model to integrate cross-modal features. However, these methods allocate most of the computation to visual encoding, while light computation on modeling modality inter- actions. Moreover, they take cross-modal fu- sion for depth features, which ignores a fine- grained alignment between text and image. Recently, vision-language pre-trained models (VLPM), encapsulating rich cross-modal align- ment knowledge, have advanced in most multi- modal tasks. In this work, we propose a novel model for TGII by improving cross-modal alignment (CMA). CMA model consists of a VLPM as a vision-language encoder, an image generator and global-local discriminators. To explore cross-modal alignment knowledge for image restoration, we introduce cross-modal alignment distillation and in-sample distribu- tion distillation. In addition, we employ ad- versarial training to enhance the model to fill the missing region in complicated structures effectively. Experiments are conducted on two popular vision-language datasets. Results show that our model achieves state-of-the-art perfor- mance compared with other strong competitors. 1 Introduction Text-guided image inpainting (TGII), involving computer vision (CV) and natural language pro- cessing (NLP), aims to restore visual content for a missing area in a damaged image based on a given text (Zhang et al., 2020a). With the development of CV and NLP, it plays an essential role in many real-world applications, such as image editing (Zhu et al., 2020), damaged image restoration (Liu et al., 2019), and image rendering (Kirillov et al., 2020). Therefore, it has become one of the most crucial areas in CV and NLP. ∗Corresponding author. Visual Encode Textual Encode Modality Interaction Text Image Modality Interaction Textual Encode Text Image Visual Encode Figure 1: Different categories of vision-and-language models. left: most of the computation on visual en- coding; right: most of the computation on modeling modality interactions. Existing methods (Zhang et al., 2020b; Lin et al., 2020; Wu et al., 2021) adopt an encoder- decoder framework as their backbone with a vision- language fusion module to introduce textual infor- mation. These methods use separate encoders for images and texts, heavier on the former. Then, the vision-language fusion module is used to integrate the features from the two modalities through a sim- ple similarity calculation of features, or shallow attention layers (Vaswani et al., 2017), as shown on the left of Figure 1. Most of the computation of these methods on visual encoding, while tex- tual features only serve as a complement to deep visual features, which ignores the importance of deep interaction of multimodal information. More- over, these methods share a common drawback: they do not perform well for …

Content truncated to 500 words