Guiding Visual Question Generation

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1640 - 1654 July 10-15, 2022 ©2022 Association for Computational Linguistics Guiding Visual Question Generation Nihir Vedd, Zixu Wang, Marek Rei, Yishu Miao, and Lucia Specia Imperial College London {n.vedd19, zixu.wang, marek.rei, y.miao20, l.specia}@imperial.ac.uk https://github.com/nihirv/guiding-vqg Abstract In traditional Visual Question Generation (VQG), most images have multiple concepts (e.g. objects and categories) for which a ques- tion could be generated, but models are trained to mimic an arbitrary choice of concept as given in their training data. This makes training dif- ficult and also poses issues for evaluation – multiple valid questions exist for most images but only one or a few are captured by the hu- man references. We present Guiding Visual Question Generation - a variant of VQG which conditions the question generator on categori- cal information based on expectations on the type of question and the objects it should ex- plore. We propose two variant families: (i) an explicitly guided model that enables an actor (human or automated) to select which objects and categories to generate a question for; and (ii) 2 types of implicitly guided models that learn which objects and categories to condition on, based on discrete variables. The proposed models are evaluated on an answer-category augmented VQA dataset and our quantitative results show a substantial improvement over the current state of the art (over 9 BLEU-4 increase). Human evaluation validates that guidance helps the generation of questions that are grammatically coherent and relevant to the given image and objects. 1 Introduction In the last few years, the AI research community has witnessed a surge in multimodal tasks such as Visual Question Answering (VQA) (Antol et al., 2015; Anderson et al., 2018), Multimodal Machine Translation (Specia et al., 2016; Elliott et al., 2017; Barrault et al., 2018; Caglayan et al., 2019), and Image Captioning (IC) (Vinyals et al., 2015; Karpa- thy and Fei-Fei, 2015; Xu et al., 2015). Visual Question Generation (VQG) (Zhang et al., 2016; Krishna et al., 2019; Li et al., 2018), a multimodal task which aims to generate a question given an image, remains relatively under-researched despite the popularity of its textual counterpart. Through- out the sparse literature in this domain, different approaches have augmented and/or incorporated extra information as input. For example, Pan et al. (2019) emphasised that providing the ground truth answer to a target question is beneficial in generat- ing a non-generic question. Krishna et al. (2019) pointed out that requiring an answer to generate questions violates a realistic scenario. Instead, they proposed a latent variable model using answer cate- gories to help generate the corresponding questions. Recently, Scialom et al. (2020) incorporated a pre- trained language model with object features and image captions for question generation. In this work, we explore VQG from the perspec- tive of ‘guiding’ a question generator. Guiding has shown success in image captioning (Zheng et al. (2018) and Ng et al. (2020)), and in …

Content truncated to 500 words