Delving Deeper into Cross-lingual Visual Question Answering

Findings of the Association for Computational Linguistics: EACL 2023, pages 2453–2468 May 2-6, 2023 ©2023 Association for Computational Linguistics Delving Deeper into Cross-lingual Visual Question Answering Chen Cecilia Liu1, Jonas Pfeiffer1, Anna Korhonen2, Ivan Vuli´c2, Iryna Gurevych1 1Ubiquitous Knowledge Processing Lab, Department of Computer Science and Hessian Center for AI (hessian.AI), Technical University of Darmstadt 2Language Technology Lab, University of Cambridge www.ukp.tu-darmstadt.de Abstract Visual question answering (VQA) is one of the crucial vision-and-language tasks. Yet, existing VQA research has mostly focused on the En- glish language, due to a lack of suitable evalua- tion resources. Previous work on cross-lingual VQA has reported poor zero-shot transfer per- formance of current multilingual multimodal Transformers with large gaps to monolingual performance, without any deeper analysis. In this work, we delve deeper into the different aspects of cross-lingual VQA, aiming to un- derstand the impact of 1) modeling methods and choices, including architecture, inductive bias, fine-tuning; 2) learning biases: including question types and modality biases in cross- lingual setups. The key results of our analysis are: 1) We show that simple modifications to the standard training setup can substantially re- duce the transfer gap to monolingual English performance, yielding +10 accuracy points over existing methods. 2) We analyze cross-lingual VQA across different question types of vary- ing complexity for different multilingual mul- timodal Transformers, and identify question types that are the most difficult to improve on. 3) We provide an analysis of modality biases present in training data and models, revealing why zero-shot performance gaps remain for certain question types and languages. 1 Introduction The lack of multilingual resources has hindered the development and evaluation of Visual Ques- tion Answering (VQA) methods beyond the En- glish language until recently. A rise in inter- est in creating multilingual Vision-and-Language (V&L) resources has inspired more research in this area (Srinivasan et al., 2021; Su et al., 2021; Liu et al., 2021a; Pfeiffer et al., 2022; Wang et al., 2022; Bugliarello et al., 2022, inter alia). Large Transformer-based models pretrained on images and text in multiple different languages have been proven as a viable vehicle for the development of multilingual V&L task architectures through trans- fer learning, but such models are still few and far between (M3P, UC2; Ni et al., 2021; Zhou et al., 2021). Large decreases in task performance be- tween monolingual and (zero-shot) cross-lingual transfer setups have been measured and reported, among other multilingual V&L tasks, in VQA (Pfeiffer et al., 2022). Yet, the reasons for such low results in this pivotal V&L task have not been investigated in depth. In this work, we aim to shed new light on the cross-lingual performance gap of cross-lingual VQA models from multiple angles. To the best of our knowledge, we are the first to provide a com- prehensive analysis of multilingual VQA, with a focus on cross-lingual transfer. We first assess and discuss the impact of model- ing methods and choices on the final cross-lingual VQA performance, aiming to mitigate the present performance gap. This includes …

Content truncated to 500 words