Mind Your Outliers! Investigating the Negative Impact of Outliers on Active Learning for Visual Question Answering

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 7265–7281 August 1–6, 2021. ©2021 Association for Computational Linguistics 7265 Mind Your Outliers! Investigating the Negative Impact of Outliers on Active Learning for Visual Question Answering Siddharth Karamcheti Ranjay Krishna Li Fei-Fei Christopher D. Manning Department of Computer Science, Stanford University {skaramcheti, ranjaykrishna, feifeili, manning}@cs.stanford.edu Abstract Active learning promises to alleviate the mas- sive data needs of supervised machine learn- ing: it has successfully improved sample ef- ficiency by an order of magnitude on tradi- tional tasks like topic classification and object recognition. However, we uncover a striking contrast to this promise: across 5 models and 4 datasets on the task of visual question an- swering, a wide variety of active learning ap- proaches fail to outperform random selection. To understand this discrepancy, we profile 8 active learning methods on a per-example ba- sis, and identify the problem as collective out- liers – groups of examples that active learning methods prefer to acquire but models fail to learn (e.g., questions that ask about text in im- ages or require external knowledge). Through systematic ablation experiments and qualita- tive visualizations, we verify that collective outliers are a general phenomenon responsi- ble for degrading pool-based active learning. Notably, we show that active learning sample efficiency increases significantly as the num- ber of collective outliers in the active learning pool decreases. We conclude with a discussion and prescriptive recommendations for mitigat- ing the effects of these outliers in future work. 1 Introduction Today, language-equipped vision systems such as VizWiz, TapTapSee, BeMyEyes, and CamFind are actively being deployed across a broad spectrum of users.1 As underlying methods improve, these systems will be expected to operate over diverse vi- sual environments and understand myriad language inputs (Bigham et al., 2010; Tellex et al., 2011; Mei et al., 2016; Zhu et al., 2017; Anderson et al., 2018b; Park et al., 2019). Visual Question Answer- ing (VQA), the task of answering questions about 1Applications can be found at https://vizwiz.org/, https://taptapsee.com/, https://www.bemyeyes.com/, and https://camfindapp.com/ Q: What sport is she playing? A: tennis Expects models to read text: Q: What is the word on the wall? A: rice Requires external knowledge: Q: What is the symbol on the hood often associated with? A: Pirates Collective outliers Dataset Map for Bottom-Up Top-Down Model on VQA 2 Useful data Q: What is the person holding? A: surfboard Figure 1: We systematically evaluate active learning on VQA datasets and isolate their inability to perform better than random sampling due to the presence of col- lective outliers. Active learning methods prefer to ac- quire these outliers, which are hard and often impos- sible for models to learn. We show that Dataset Maps, like the one shown here, can heuristically identify these collective outliers as examples assigned low model con- fidence and prediction variability during training. visual inputs, is a popular benchmark used to eval- uate progress towards such open-ended systems …

Content truncated to 500 words