Transformer Based Multi-Source Domain Adaptation

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 878–892, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 878 MUTANT: A Training Paradigm for Out-of-Distribution Generalization in Visual Question Answering Tejas Gokhale ∗and Pratyay Banerjee ∗ and Chitta Baral and Yezhou Yang Arizona State University tgokhale,pbanerj6,chitta,yz.yang@asu.edu Abstract While progress has been made on the visual question answering leaderboards, models of- ten utilize spurious correlations and priors in datasets under the i.i.d. setting. As such, evaluation on out-of-distribution (OOD) test samples has emerged as a proxy for general- ization. In this paper, we present MUTANT, a training paradigm that exposes the model to perceptually similar, yet semantically dis- tinct mutations of the input, to improve OOD generalization, such as the VQA-CP chal- lenge. Under this paradigm, models utilize a consistency-constrained training objective to understand the effect of semantic changes in input (question-image pair) on the output (an- swer). Unlike existing methods on VQA-CP, MUTANT does not rely on the knowledge about the nature of train and test answer distri- butions. MUTANT establishes a new state-of- the-art accuracy on VQA-CP with a 10.57% improvement. Our work opens up avenues for the use of semantic input mutations for OOD generalization in question answering. 1 Introduction Availability of large-scale datasets has enabled the use of statistical machine learning in vision and language understanding, and has lead to significant advances. However, the commonly used evalu- ation criterion is the performance of models on test-samples drawn from the same distribution as the training dataset, which cannot be a measure of generalization. Training under this “indepen- dent and identically distributed” (i.i.d.) setting can drive decision making to be highly influenced by dataset biases and spurious correlations as shown in both natural language inference (Kaushik and Lipton, 2018; Poliak et al., 2018; McCoy et al., 2019) and visual question answering (Goyal et al., ∗Equal Contribution What is the color of the frisbee? What is the child doing? GREEN → PINK PLAYING → ACTION IMAGE MUTATION IMAGE MUTATION QUESTION MUTATION QUESTION MUTATION How many bottles? Is the meal not healthy? How many bins? Is the meal healthy? TWO → NONE YES → NO IMAGE MUTATION QUESTION MUTATION Figure 1: Illustration of the mutant samples. The in- put mutation, either by manipulating the image or the question, results in a change in the answer. 2017; Agrawal et al., 2018a; Selvaraju et al., 2020). As such, evaluation on out-of-distribution (OOD) samples has emerged as a metric for generalization. Visual question answering (VQA) (Antol et al., 2015) is a task at the crucial intersection of vision and language. The aim of VQA models is to pro- vide an answer, given an input image and a ques- tion about it. Large datasets (Antol et al., 2015) have been extensively used for developing VQA models. However over-reliance on datasets can cause models to learn spurious correlations such as linguistic priors (Agrawal et al., 2018a) that are specific to certain datasets and do not generalize to “Out-of-Distribution” (OOD) samples, as shown …

Content truncated to 500 words