Being Negative but Constructively: Lessons Learnt from Creating Better Visual Question Answering Datasets

Proceedings of NAACL-HLT 2018, pages 431–441 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Being Negative but Constructively: Lessons Learnt from Creating Better Visual Question Answering Datasets Wei-Lun Chao∗, Hexiang Hu∗, Fei Sha University of Southern California Los Angeles, California, USA weilunchao760414@gmail.com, hexiang.frank.hu@gmail.com, feisha@usc.edu Abstract Visual question answering (Visual QA) has at- tracted a lot of attention lately, seen essentially as a form of (visual) Turing test that artificial intelligence should strive to achieve. In this paper, we study a crucial component of this task: how can we design good datasets for the task? We focus on the design of multiple- choice based datasets where the learner has to select the right answer from a set of candi- date ones including the target (i.e. the correct one) and the decoys (i.e. the incorrect ones). Through careful analysis of the results attained by state-of-the-art learning models and human annotators on existing datasets, we show that the design of the decoy answers has a sig- nificant impact on how and what the learn- ing models learn from the datasets. In par- ticular, the resulting learner can ignore the vi- sual information, the question, or both while still doing well on the task. Inspired by this, we propose automatic procedures to remedy such design deficiencies. We apply the pro- cedures to re-construct decoy answers for two popular Visual QA datasets as well as to cre- ate a new Visual QA dataset from the Vi- sual Genome project, resulting in the largest dataset for this task. Extensive empirical stud- ies show that the design deficiencies have been alleviated in the remedied datasets and the performance on them is likely a more faith- ful indicator of the difference among learn- ing models. The datasets are released and publicly available via http://www.teds. usc.edu/website_vqa/. 1 Introduction Multimodal information processing tasks such as image captioning (Farhadi et al., 2010; Ordonez et al., 2011; Xu et al., 2015) and visual question answering (Visual QA) (Antol et al., 2015) have ∗Equal contributions Question: What vehicle is pictured? Image only Unresolvable (IoU) a.#Overcast. (0.5455) b.#Daytime.# (0.4941) c.#A#building. (0.4829) d.#A#train. (0.5363) Question only Unresolvable (QoU) a.#A#bicycle. (0.2813) b.#A#truck. (0.5364)# c.#A#boat. (0.4631) d.#A#train. (0.5079) Original a.#A#car.# (0.2083) b.#A#bus.# (0.6151) c.#A#cab.# (0.5000) d.#A#train.# (0.7328) Candidate Answers: Figure 1: An illustration of how the shortcuts in the Vi- sual7W dataset (Zhu et al., 2016) should be remedied. In the original dataset, the correct answer “A train” is easily selected by a machine as it is far often used as the correct answer than the other decoy (negative) an- swers. (The numbers in the brackets are probability scores computed using eq. (2)). Our two procedures — QoU and IoU (cf. Sect. 4) — create alternative decoys such that both the correct answer and the decoys are highly likely by examining either the image or the ques- tion alone. In these cases, machines make mistakes un- less they consider all information together. Thus, the alternative decoys suggested our procedures are better designed …

Content truncated to 500 words