Visual Question Generation in Bengali

Proceedings of the Workshop on Multimodal, Multilingual Natural Language Generation and Multilingual WebNLG Challenge (MM-NLG 2023), pages 10–19 September 12, 2023 ©2023 Association for Computational Linguistics Visual Question Generation in Bengali Mahmud Hasan, Labiba Islam, Jannatul Ferdous Ruma Tasmiah Tahsin Mayeesha and Rashedur M. Rahman North South University {mahmud.hasan03,labiba.islam,jannatul.ruma, tasmiah.tahsin,rashedur.rahman}@northsouth.edu Abstract The task of Visual Question Generation (VQG) is to generate human-like questions relevant to the given image. As VQG is an emerging re- search field, existing works tend to focus only on resource-rich language such as English due to the availability of datasets. In this paper, we propose the first Bengali Visual Question Gen- eration task and develop a novel transformer- based encoder-decoder architecture that gener- ates questions in Bengali when given an im- age. We propose multiple variants of models - (i) image-only: baseline model of generating questions from images without additional infor- mation, (ii) image-category and image-answer- category: guided VQG where we condition the model to generate questions based on the answer and the category of expected question. These models are trained and evaluated on the translated VQAv2.0 dataset. Our quantitative and qualitative results establish the first state of the art models for VQG task in Bengali and demonstrate that our models are capable of generating grammatically correct and relevant questions. Our quantitative results show that our image-cat model achieves a BLUE-1 score of 33.12 and BLEU-3 score of 7.56 which is the highest of the other two variants. We also perform a human evaluation to assess the qual- ity of the generation tasks. Human evaluation suggests that image-cat model is capable of generating goal-driven and attribute-specific questions and also stays relevant to the cor- responding image. 1 Introduction Visual Question Generation (VQG) is an emerg- ing research field in both Computer Vision and Natural Language Processing. The task of VQG simply uses an image and other side information (e.g. answers or answer categories) as input and generates meaningful questions related to the im- age. Tasks like cross-modal Visual Question An- swering (VQA) (Antol et al., 2015; Cadene et al., Figure 1: Examples of Bengali VQG Predictions with category of answers as additional information. 2019; Peng et al., 2019; Jiang et al., 2020; Guo et al., 2022), Video Captioning (VC) (Chen et al., 2019), Image Captioning (IC) (Vinyals et al., 2015; Karpathy and Fei-Fei, 2017; Xu et al., 2015), and Multimodal Machine Translation (Specia et al., 2016; Elliott et al., 2017; Barrault et al., 2018; Caglayan et al., 2019) are the recent advances in the AI community. While the majority of visuo- lingual tasks tend to focus on VQA, a few recent approaches have been proposed, focusing on the under-researched multi-modal task of VQG. VQG is a more creative and particularly challenging prob- lem than VQA, because the generated questions need to be relevant, semantically coherent and com- prehensible to the diverse contents of the given image. Existing studies on Visual Question Generation (VQG) have been primarily focused on languages that have ample resources, such as English. …

Content truncated to 500 words