Multimodal Learning for Accurate Visual Question Answering: An Attention-Based Approach

Proceedings of Recent Advances in Natural Language Processing, pages 179–186 Varna, Sep 4–6, 2023 https://doi.org/10.26615/978-954-452-092-2_020 179 Multimodal Learning for Accurate Visual Question Answering: An Attention-based Approach Jishnu Bhardwaj, Anurag Balakrishnan, Satyam Pathak, Ishan Unnarkar, Aniruddha Gawande, and Benyamin Ahmadnia Department of Computer Engineering and Computer Science California State University, Long Beach, United States jishnu.bhardwaj01@student.csulb.edu, anurag.balakrishnan01@student.csulb.edu, satyam.pathak01@student.csulb.edu, ishan.unnarkar01@student.csulb.edu, aniruddharajendra.gawande01@student.csulb.edu, benyamin.ahmadnia@csulb.edu Abstract This paper proposes an open-ended task for Visual Question Answering (VQA) that lever- ages the InceptionV3 Object Detection model and an attention-based Long Short-Term Mem- ory (LSTM) network for question answering. Our proposed model provides accurate natural language answers to questions about an im- age, including those that require understanding contextual information and background details. Our findings demonstrate that the proposed ap- proach can achieve high accuracy, even with complex and varied visual information. The proposed method can contribute to developing more advanced vision systems that can process and interpret visual information like humans. 1 Introduction As Computer Vision research moves beyond “buck- eted” identification and toward resolving multi- modal problems, language and visual problems like picture captioning and Visual Question Answering (VQA) have become prominent (Fang et al., 2015). Issues in the nexus of vision and language are com- plex due to the complicated compositional structure of language (Fukui et al., 2016) (Kafle and Kanan, 2017). However, recent research has shown that language can also provide a strong prior that can lead to good performance on the surface even when the underlying models do not fully comprehend the visual information. Our approach to solving the VQA problem in- volves the development of three distinct models, each with its strengths and limitations: The first model is a simple baseline model that utilizes a pre-trained Convolutional Neural Network (CNN) and Recurrent Neural Network (RNN) architec- tures to extract visual and textual features from the input image and question, respectively. These fea- tures are then concatenated and fed into a simple feed-forward neural network that outputs the final answer. The second is an attention-based model that builds upon the baseline model by incorporat- ing attention mechanisms to selectively focus on relevant parts of the image and question during the feature extraction process. This allows the model to attend to different regions of the image and words in the question, depending on their relevance to the answer. The third model is a more complex, multi-modal transformer-based model that uses a pre-trained transformer architecture to extract vi- sual and textual features from the input image and question. The transformer model incorporates self- attention mechanisms that allow it to learn the re- lationships between different input parts and se- lectively attend to the most relevant information. This model also incorporates a Visual-Linguistic Transformer (ViLT) module that learns joint repre- sentations of both the image and question, allowing for a more seamless integration of visual and tex- tual information. The experimental results show that our models employ different approaches to fea- ture extraction and utilize various neural network architectures to tackle the VQA problem. 2 Related …

Content truncated to 500 words