Bi-directional Cross-Attention Network on Vietnamese Visual Question Answering

Bi-directional Cross-Attention Network on Vietnamese Visual Question Answering Duy-Minh Nguyen-Tran Faculty of Information Technology, University of Science, Vietnam Vietnam National University, Ho Chi Minh city, Vietnam 20c11041@student.hcmus.edu.vn Tung Le Faculty of Information Technology, University of Science, Vietnam Vietnam National University, Ho Chi Minh city, Vietnam lttung@fit.hcmus.edu.vn Minh Le Nguyen Japan Advanced Institute of Science and Technology, Nomi, Ishikawa, Japan nguyenml@jaist.ac.jp Huy Tien Nguyen Faculty of Information Technology, University of Science, Vietnam Vietnam National University, Ho Chi Minh city, Vietnam ntienhuy@fit.hcmus.edu.vn Corresponding author: Huy Tien Nguyen ntienhuy@fit.hcmus.edu.vn Abstract Visual Question Answering (VQA) has arisen in recent public interest thanks to its appli- cability in many different fields. However, it requires understanding the combination of pictures and questions, which is highly chal- lenging in both vision and language process- ing. Many previous works have achieved re- markable results to address this problem in many different languages. However, in the Vietnamese language, the VQA problem has not made significant progress due to the lack of data and fundamental systems. Therefore, we propose a model specifically designed and optimized for the Vietnamese Visual Ques- tion Answering problem. Our model lever- ages the strength of pre-trained models as well as presents Bi-directional Cross-attention ar- chitecture to learn visual and textual features more effectively. Through experimental re- sults and ablation studies, the proposed ap- proach obtains promising results against the existing models for Vietnamese on the ViVQA dataset. 1 Introduction Together with the remarkable development in Com- puter Vision and Natural Language Processing, many problems requiring the combination of both images and languages were raised such as Image Captioning (Wang et al., 2022; Hu et al., 2021), Vi- sual Question Answering (VQA) (Le et al., 2021a; Le et al., 2020), Visual Question Classification (Le. et al., 2022), etc. Among them, VQA is a poten- tial area receiving a lot of attention in both research and industry. Furthermore, the recent approaches in VQA also achieved promising results. The goal of most VQA systems is to digest the content of a given image and answer the related questions. Com- pared to other tasks like Image Captioning and Vi- sual Question Classification, VQA requires a deeper understanding of visual and textual inputs to give ap- propriate answers. In practice, VQA can be applied to various scenarios such as human-machine inter- action, medical assistance, and automatic customer service or recommendation. Undoubtedly, the VQA model plays an important role in the cutting edge of multi-modal approaches between images and languages. In recent years, many VQA approaches have been introduced and achieved promising results such as LXMERT (Tan and Bansal, 2019), SIMVL (Wang et al., 2021), OS- CAR (Li et al., 2020). However, current VQA sys- tems are almost addressed in English, Japanese, Chi- nese, and a few other languages. However, VQA for Vietnamese has not been strongly developed due to data limitations. Therefore, we introduce an op- timized and fine-tuned VQA model specifically for the Vietnamese language. Besides, we propose bi- directional cross-attention architecture by adjusting the multi-head Attention in Transformer …

Content truncated to 500 words