Understanding Attention for Vision-and-Language Tasks

Proceedings of the 29th International Conference on Computational Linguistics, pages 3438–3453 October 12–17, 2022. 3438 Understanding Attention for Vision-and-Language Tasks Feiqi Cao1, Soyeon Caren Han1,2, Siqu Long1, Changwei Xu1, Josiah Poon1 1School of Computer Science, The University of Sydney, Australia 2School of Physics, Maths and Computing, The University of Western Australia, Australia {fcao0492, slon6753, chxu7327}@uni.sydney.edu.au {caren.han, josiah.poon}@sydney.edu.au Abstract Attention mechanism has been used as an important component across Vision-and- Language(VL) tasks in order to bridge the se- mantic gap between visual and textual features. While attention has been widely used in VL tasks, it has not been examined the capabil- ity of different attention alignment calculation in bridging the semantic gap between visual and textual clues. In this research, we con- duct a comprehensive analysis on understand- ing the role of attention alignment by looking into the attention score calculation methods and check how it actually represents the visual re- gion’s and textual token’s significance for the global assessment. We also analyse the condi- tions which attention score calculation mech- anism would be more (or less) interpretable, and which may impact the model performance on three different VL tasks, including visual question answering, text-to-image generation, text-and-image matching (both sentence and image retrieval). Our analysis is the first of its kind and provides useful insights of the im- portance of each attention alignment score cal- culation when applied at the training phase of VL tasks, commonly ignored in attention-based cross modal models, and/or pretrained models. Our code is available at: https://github. com/adlnlp/Attention_VL 1 Introduction The relative maturity and flexibility of deep learn- ing allow us to build upon the success of com- puter vision and natural language processing to face many complex and multimodal Vision-and- Language (VL) tasks, such as Visual Question An- swering (VQA), Text-and-Image Matching (T&I Match), or Text-to-Image Generation (T2I Gen). For these VL tasks, it is crucial to effectively align the multimodal information in both visual and lin- guistic domains. For example, to pick the right answer in VQA, the model should empower infor- mation from the input image, together with aligning the linguistic meanings with visual clues. Attention mechanism (Bahdanau et al., 2015; Lu- ong et al., 2015a) has been used as an important component across a wide range of VL models; from the early-stage attention-based fusion VL models (Shih et al., 2016; Wang et al., 2019; Xu et al., 2018a; Yang et al., 2016) to the recent VL mul- timodal transformer-based pretrained models (Hu et al., 2021; Li et al., 2020b; Lu et al., 2019; Long et al., 2022a). Those attention-based VL models mainly focus on 1) exploring new features to repre- sent visual and linguistic information as an input of attention layer, 2) deciding the position or the num- ber of attentions in the model, or 3) investigating the interpretability of attention distribution on VL tasks by emphasising the specific image regions or textual tokens (Luo et al., 2021). While such approaches and investigations have resulted in interesting findings in different aspects of VL …

Content truncated to 500 words