xGQA: Cross-Lingual Visual Question Answering

Findings of the Association for Computational Linguistics: ACL 2022, pages 2497 - 2511 May 22-27, 2022 c⃝2022 Association for Computational Linguistics xGQA: Cross-Lingual Visual Question Answering Jonas Pfeiffer1, Gregor Geigle1, Aishwarya Kamath2, Jan-Martin O. Steitz3 Stefan Roth3, Ivan Vuli´c4, Iryna Gurevych1 1Ubiquitous Knowledge Processing Lab, Technical University of Darmstadt 2Center for Data Science, New York University 3Visual Inference Lab, Technical University of Darmstadt 4Language Technology Lab, University of Cambridge Abstract Recent advances in multimodal vision and lan- guage modeling have predominantly focused on the English language, mostly due to the lack of multilingual multimodal datasets to steer modeling efforts. In this work, we ad- dress this gap and provide xGQA, a new mul- tilingual evaluation benchmark for the visual question answering task. We extend the es- tablished English GQA dataset (Hudson and Manning, 2019) to 7 typologically diverse lan- guages, enabling us to detect and explore cru- cial challenges in cross-lingual visual ques- tion answering. We further propose new adapter-based approaches to adapt multimodal transformer-based models to become multilin- gual, and—vice versa—multilingual models to become multimodal. Our proposed meth- ods outperform current state-of-the-art multi- lingual multimodal models (e.g., M3P) in zero- shot cross-lingual settings, but the accuracy remains low across the board; a performance drop of around 38 accuracy points in target lan- guages showcases the difficulty of zero-shot cross-lingual transfer for this task. Our results suggest that simple cross-lingual transfer of multimodal models yields latent multilingual multimodal misalignment, calling for more so- phisticated methods for vision and multilin- gual language modeling.1 1 Introduction Transformer-based architectures (Vaswani et al., 2017) have become ubiquitous in NLP (Devlin et al., 2019; Liu et al., 2019; Conneau et al., 2020, inter alia) and in computer vision (CV) (Carion et al., 2020; Dosovitskiy et al., 2021), offering un- matched task performance. Having a shared archi- tecture for multiple modalities opened up possibil- ities for effective fusion of information, yielding impressive performance gains across various mul- timodal tasks such as image captioning, phrase 1The xGQA dataset is available online at: https:// github.com/Adapter-Hub/xGQA. En De Bn Pt Ru Zh Ko Id Who is flying through the sky? Wer fliegt durch die Luft? আকােশর মেধধ্যে িদেয় �ক উড়েছ? Quem está voando pelo céu? Кто летает по небу? 谁在天空中飞过？ 누가 하늘을 날고 있습니까? Siapa yang sedang terbang melintasi langit? Skateboarder Figure 1: Example taken from the xGQA dataset with the same question uttered in 8 languages. grounding, visual question answering, referring ex- pression comprehension and image-text retrieval (Lu et al., 2019; Tan and Bansal, 2019; Li et al., 2020b; Zhang et al., 2021; Ni et al., 2021; Kamath et al., 2021; Miech et al., 2021; Frank et al., 2021; Bugliarello et al., 2021; Radford et al., 2021; Jia et al., 2021; Eichenberg et al., 2021; Singh et al., 2021; Fu et al., 2021; Yang et al., 2021; Yuan et al., 2021; Wang et al., 2021a; Li et al., 2021; Geigle et al., 2022, inter alia). Yet, progress in this area has been limited mostly to the English language, …

Content truncated to 500 words