Can Pre-training help VQA with Lexical Variations?

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2863–2868 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 2863 Can Pre-training help VQA with Lexical Variations? Shailza Jolly TU Kaiserslautern, Germany DFKI GmbH, Germany shailza.jolly@dfki.de Shubham Kapoor1 Amazon Research, Germany kapooshu@amazon.com Abstract Rephrasings or paraphrases are sentences with similar meanings expressed in different ways. Visual Question Answering (VQA) models are closing the gap with the oracle performance for datasets like VQA2.0. However, these models fail to perform well on rephrasings of a question, which raises some important ques- tions like Are these models robust towards lin- guistic variations? Is it the architecture or the dataset that we need to optimize? In this paper, we analyzed VQA models in the space of para- phrasing. We explored the role of language & cross-modal pre-training to investigate the ro- bustness of VQA models towards lexical vari- ations. Our experiments find that pre-trained language encoders generate efficient represen- tations of question rephrasings, which help VQA models correctly infer these samples. We empirically determine why pre-training language encoders improve lexical robustness. Finally, we observe that although pre-training all VQA components obtain state-of-the-art results on the VQA-Rephrasings dataset, it still fails to completely close the performance gap between original and rephrasing validation splits. 1 Introduction Visual Question Answering (VQA) (Antol et al., 2015) is an image conditioned question answering task which has gained immense popularity in vision & language community. Since the introduction of the VQA challenge2, there has been significant progress in the field of VQA, where new model architectures and training techniques are closing the gap between the model and oracle accuracy on benchmarking datasets like VQA2.0 (Goyal et al., 2017). A majority of models obtained higher gains 1The work was done prior to joining Amazon. 2https://visualqa.org/challenge.html Figure 1: Example from VQA-Rephrasings dataset (Shah et al., 2019). The answers are obtained using Pythia (Jiang et al., 2018) where green text refers to correct answer and red text refers to wrong answer. by introducing semantically rich visual features (Anderson et al., 2018), efficient attention schemes (Lu et al., 2016; Yang et al., 2016), and advance multimodal fusion techniques (Fukui et al., 2016; Yu et al., 2017). However, to deploy these state-of-the-art VQA models into real-world settings, the models must be robust to linguistic variations that originate from interactions with real users. Recently, Shah et al. (2019) showed that state-of-the-art VQA models (Jiang et al., 2018; Kim et al., 2018) are extremely sensitive to the lexical variations which result in a significant performance drop on the VQA test datasets when the questions are replaced with their rephrases. Figure 1 shows the shift in confidence scores of answers for a rephrasing of the original question. To handle these scenarios, they provided a model-agnostic cyclic-consistency (CC) approach that generates question rephrases on the fly during training, which makes the underlying VQA model lexically robust. The best-reported model with their approach achieves 56.59% VQA accuracy on ques- tion rephrasings. Nevertheless, all the models that Shah et …

Content truncated to 500 words