Towards a Unified Model for Generating Answers and Explanations in Visual Question Answering

Findings of the Association for Computational Linguistics: EACL 2023, pages 1693–1705 May 2-6, 2023 ©2023 Association for Computational Linguistics Towards a Unified Model for Generating Answers and Explanations in Visual Question Answering Chenxi Whitehouse, Tillman Weyde, Pranava Madhyastha City, University of London {chenxi.whitehouse, t.e.weyde, pranava.madhyastha}@city.ac.uk Abstract The field of visual question answering (VQA) has recently seen a surge in research focused on providing explanations for predicted answers. However, current systems mostly rely on sep- arate models to predict answers and generate explanations, leading to less grounded and fre- quently inconsistent results. To address this, we propose a multitask learning approach towards a Unified Model for Answer and Explanation generation (UMAE). Our approach involves the addition of artificial prompt tokens to train- ing data and fine-tuning a multimodal encoder- decoder model on a variety of VQA-related tasks. In our experiments, UMAE models sur- pass the prior state-of-the-art answer accuracy on A-OKVQA by 10∼15%, show competitive results on OK-VQA, achieve new state-of-the- art explanation scores on A-OKVQA and VCR, and demonstrate promising out-of-domain per- formance on VQA-X.1 1 Introduction Contemporary models for visual question answer- ing (VQA) and commonsense reasoning are typ- ically trained discriminatively to select the best answers from Multiple-Choice questions or to clas- sify single-word answers to a predetermined vocab- ulary (e.g. Anderson et al., 2018). Such settings of- ten lead to limitations such as encouraging models to find superficial correlations (Ye and Kovashka, 2021) or penalising model performance even when the answers are plausible (e.g. synonyms and multi- word expressions, and morphological variations are not considered correct). Most current explanation generation models are trained independently of the QA model and the explanations are usually gener- ated after the QA model has provided an answer. As a result, these explanation models lack access to the process that generated the answer and thus 1Code is available at: https://github.com/ chenxwh/UMAE. the grounding of the explanation is limited to the answer text. We posit that a unified model that simultaneously performs answer prediction and explanation genera- tion is a more effective and consistent approach for VQA. Generative models, such as GPT-3 (Brown et al., 2020), T5 (Raffel et al., 2020), or OFA (Wang et al., 2022a), have been shown to be successful at rapidly adapting to downstream tasks and gener- ating high-quality open-ended text, and hence are suitable candidates for this unified approach. We propose a multitask learning approach for multimodal transformer-based encoder-decoder models, towards a United Model for Answer and Explanation generation (UMAE). In addition to the current trend of separate answer prediction and explanation generation based on the answers, our approach adds the capability of jointly generating answers and explanations together. Inspired by the success of artificial prompt tokens in Neural Ma- chine Translation (NMT) (Johnson et al., 2017), we extend and demonstrate the efficacy of the ar- tificial prompt-based method for VQA in a mul- titask setup. We augment training instances with artificial prompt tokens, enabling the model to dis- tinguish different tasks while learning shared se- mantic features. …

Content truncated to 500 words