Cross-document Event Identity via Dense Annotation

Proceedings of the 25th Conference on Computational Natural Language Learning (CoNLL), pages 27–43 November 10–11, 2021. ©2021 Association for Computational Linguistics 27 VQA-MHUG: A Gaze Dataset to Study Multimodal Neural Attention in Visual Question Answering Ekta Sood1, Fabian Kögel1, Florian Strohm1, Prajit Dhar2, Andreas Bulling1 1University of Stuttgart, Institute for Visualization and Interactive Systems (VIS), Germany 2University of Groningen, Center for Language and Cognition (CLCG), the Netherlands {ekta.sood,fabian.koegel,florian.strohm,andreas.bulling}@vis.uni-stuttgart.de p.dhar@rug.nl Abstract We present VQA-MHUG – a novel 49- participant dataset of multimodal human gaze on both images and questions during visual question answering (VQA) collected using a high-speed eye tracker. We use our dataset to analyze the similarity between human and neu- ral attentive strategies learned by five state-of- the-art VQA models: Modular Co-Attention Network (MCAN) with either grid or region features, Pythia, Bilinear Attention Network (BAN), and the Multimodal Factorized Bilin- ear Pooling Network (MFB). While prior work has focused on studying the image modality, our analyses show – for the first time – that for all models, higher correlation with human attention on text is a significant predictor of VQA performance. This finding points at a po- tential for improving VQA performance and, at the same time, calls for further research on neural text attention mechanisms and their in- tegration into architectures for vision and lan- guage tasks, including but potentially also be- yond VQA. 1 Introduction Visual question answering (VQA) has gained pop- ularity as a practically-useful and challenging task at the intersection of natural language processing (NLP) and computer vision (CV) (Antol et al., 2015). The key challenge in VQA is to develop computational models that are able to reason over questions and images in order to generate answers that are well-grounded in both modalities (P. Zhang et al., 2015; Agrawal et al., 2016; Goyal et al., 2017a; Kafle et al., 2019). Attention mechanisms originally introduced in NLP for monomodal lan- guage tasks have been successfully applied to mul- timodal tasks (like VQA) and established a new state of the art (Correia and Colombini, 2021; Kim et al., 2018; Yu et al., 2019b). These advances have, in turn, triggered research into understanding the reasons for these improve- ments. A body of work has studied similarities between neural and human attention (Qiuxia et al., 2020; Yun et al., 2013; Das et al., 2016). Models seem to learn very different attention strategies and similarity to human attention might only improve performance for specific model types (Sood et al., 2020a). However, although VQA is an inherently multimodal task, all of these analyses have only focused on image attention. The most likely reason for this is that existing datasets only offer mono- modal attention on the image (Das et al., 2016; Fosco et al., 2020; Chen et al., 2020). In addition, due to the challenges involved in recording human gaze data at scale, prior works have instead used mouse data as a proxy to attention (Jiang et al., 2015). However, mouse data was shown to over- estimate some image areas (Tavakoli …

Content truncated to 500 words