Curriculum Learning Effectively Improves Low Data VQA

Curriculum Learning Effectively Improves Low Data VQA Narjes Askarian Dept. of Data Science and AI Monash University Ehsan Abbasnejad Australian Institute for Machine Learning The Univ. of Adelaide Ingrid Zukerman and Wray Buntine and Gholamreza Haffari Dept. of Data Science and AI Monash University Abstract Visual question answering (VQA) models, in particular modular ones, are commonly trained on large-scale datasets to achieve state of the art performance. However, such datasets are sometimes not available. Further, it has been shown that training these models on small datasets significantly reduces their accuracy. In this paper, we propose a curriculum-based learning (CL) regime to increase the accu- racy of VQA models trained on small datasets. Specifically, we offer three criteria to rank the samples in these datasets, and propose a training strategy for each criterion. Our re- sults show that, for small datasets, our CL ap- proach yields more accurate results than those obtained when training with no curriculum. 1 Introduction Visual question answering (VQA) models are com- monly trained on large-scale datasets to achieve the state of the art performance (Johnson et al., 2017a; Antol et al., 2015; Hudson and Manning, 2019). Modular VQA models, in particular, require large data sets for training. These models dynamically combine a number of neural networks according to a pre-specified layout (Andreas et al., 2016; John- son et al., 2017b; Yu et al., 2018) to form a new larger network that produces an answer to an input question. The layout, or program, is generated for each question on the fly. As a consequence, the ar- chitecture of the resulting network varies according to the program. Combining neural networks often leads to a wide and deep network. Training such a large-sized net- work with a varying architecture calls for a massive amount of labeled data, which is either expensive or very limited in many realistic settings. With in- sufficient data, a large and complex network can perform unsuccessfully. An example of this is our experience in training the VQA model by John- son et al. (2017b) with only 20% of the CLEVR dataset (Johnson et al., 2017a). Our results showed only 54.24% accuracy compared to the accuracy of 96.90% on the full dataset according to the au- thors’ report (Johnson et al., 2017b). Motivated by this experience, the work presented in this paper studies VQA in low data scenarios, and sheds light on the performance of current modular VQA mod- els under data scarcity conditions. To the best of our knowledge, this is the first study to investigate VQA models in low-data regime. Many approaches have been investigated to improve the performance of deep learning mod- els when training on limited data, ranging from data augmentations (Zhang et al., 2019) and pre- training (Erhan et al., 2010) to semi-supervised learning (Kingma et al., 2014) and transfer learn- ing (Raina et al., 2007). However, these works mostly deal with the scarcity of labeled data by assuming help from available unlabeled data, or by transferring knowledge from similar domains. …

Content truncated to 500 words