Building Joint Relationship Attention Network for Image-Text Generation

Proceedings of the 29th International Conference on Computational Linguistics, pages 5521–5531 October 12–17, 2022. 5521 Building Joint Relationship Attention Network for Image-Text Generation Changzhi Wang and XiaoDong Gu∗ Department of Electronic Engineering, Fudan University Shanghai, 200438, China 18110720047@fudan.edu.cn(C. Wang) xdgu@fudan.edu.cn(X. Gu) Abstract Attention based methods for image-text gen- eration often focus on visual features individ- ually, while ignoring relationship information among image features that provides important guidance for generating sentences. To alleviate this issue, in this work we propose the Joint Relationship Attention Network (JRAN) that novelly explores the relationships among the features. Specifically, different from the pre- vious relationship based approaches that only explore the single relationship in the image, our JRAN can effectively learn two relationships, the visual relationships among region features and the visual-semantic relationships between region features and semantic features, and fur- ther make a dynamic trade-off between them during outputting the relationship representa- tion. Moreover, we devise a new relationship based attention, which can adaptively focus on the output relationship representation when predicting different words. Extensive experi- ments on large-scale MSCOCO and small-scale Flickr30k datasets show that JRAN achieves state-of-the-art performance. More remarkably, JRAN achieves new 28.3% and 58.2% perfor- mance in terms of BLEU4 and CIDEr metric on Flickr30k dataset. 1 Introduction Image-text generation (i.e., image captioning) is a typical cross-modal task that connects Natural Language Processing (NLP) and Computer Vision (CV) (Tahvili et al., 2020). Its core goal is to auto- matically predict a meaningful and grammatically correct sentence, which can accurately describe the main content of images. Practical applications for this task mainly include injecting visual intelli- gence into the chatbots, searching semantic image, and helping people with visual impairments to un- derstand the visual world. However, image-text generation is still a challenging task. The main ∗Corresponding author. Figure 1: Illustration of different schemes. (a) is an example predicted by SC (base) baselines (Chen et al., 2017) that mainly uses visual features to generate sen- tence. (b) presents a more accurate sentence generated by our JRAN method which learns two relationship, i.e., visual relationship between region features and visual- semantic relationship between region features and se- mantic features (i.e., background and environment). difficulties originate from two aspect: 1) The noise and complex background in the image are likely to interfere with the generation of correct caption; 2) The interaction between features in the image is often overlooked. For difficulty 1), encouraging by the method (Wu et al., 2021b), the noise can be injected into RNN hidden states to predict the mean and stan- dard deviation, and manipulate the RNN transition states. In this way, the network robustness can be significantly enhanced and the issue can be well solved. However for difficulty 2), although some related visual attention based methods (Xu et al., 2015; Wang et al., 2016; Song et al., 2018) achieve remarkable progress, they usually focus on the im- age visual features while ignoring the relationships between them. This makes the model often difficult to generate an accurate or appropriate description …

Content truncated to 500 words