A Unified Framework for Multilingual and Code-Mixed Visual Question Answering

Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 900–913 December 4 - 7, 2020. c⃝2020 Association for Computational Linguistics 900 A Unified Framework for Multilingual and Code-Mixed Visual Question Answering Deepak Gupta‡, Pabitra Lenka†∗, Asif Ekbal‡, Pushpak Bhattacharyya‡ ‡Indian Institute of Technology Patna, India †International Institute of Information Technology Bhubaneswar, India ‡{deepak.pcs16, asif, pb}@iitp.ac.in †pabitra.lenka18@gmail.com Abstract In this paper, we propose an effective deep learning framework for multilingual and code- mixed visual question answering. The pro- posed model is capable of predicting answers from the questions in Hindi, English or Code- mixed (Hinglish: Hindi-English) languages. The majority of the existing techniques on Vi- sual Question Answering (VQA) focus on En- glish questions only. However, many applica- tions such as medical imaging, tourism, visual assistants require a multilinguality-enabled module for their widespread usages. As there is no available dataset in English-Hindi VQA, we firstly create Hindi and Code-mixed VQA datasets by exploiting the linguistic properties of these languages. We propose a robust tech- nique capable of handling the multilingual and code-mixed question to provide the answer against the visual information (image). To better encode the multilingual and code-mixed questions, we introduce a hierarchy of shared layers. We control the behaviour of these shared layers by an attention-based soft layer sharing mechanism, which learns how shared layers are applied in different ways for the dif- ferent languages of the question. Further, our model uses bi-linear attention with a residual connection to fuse the language and image fea- tures. We perform extensive evaluation and ablation studies for English, Hindi and Code- mixed VQA. The evaluation shows that the proposed multilingual model achieves state-of- the-art performance in all these settings. 1 Introduction Visual Question Answering (VQA) is a challeng- ing problem that requires complex reasoning over visual elements to provide an accurate answer to a natural language question. An efficient VQA sys- tem can be used to build an Artificial Intelligence (AI) agent which takes a natural language question ∗Work carried out during the internship at IIT Patna and predicts the decision by analyzing the complex scene(s). VQA requires language understanding, fine-grained visual processing and multiple steps of reasoning to produce the correct answer. As the existing research on VQA are mainly focused on natural language questions written in English (Antol et al., 2015; Hu et al., 2017; Fukui et al., 2016; Anderson et al., 2018; Li et al., 2018; Xu and Saenko, 2016; Shih et al., 2016), their applications are often limited. QE: What color are the trees? QH: पेड़ के क्या रंग हैं? (Trans: What color are the trees?) QCM: Trees ke kya color hain? (Trans: What color are the trees?) Answer (English): Green Answer (Hindi): हरा QE: Where is this picture? QH: यह तस्वीर कहां की है? (Trans: Where is this picture?) QCM: Yah picture kahan ki hai? (Trans: Where is this picture?) Answer: Market Answer (Hindi): बाजार Figure 1: Examples of questions …

Content truncated to 500 words