Shifting the Baseline: Single Modality Performance on Visual Navigation & QA

Proceedings of NAACL-HLT 2019, pages 1977–1983 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 1977 Shifting the Baseline: Single Modality Performance on Visual Navigation & QA Jesse Thomason Daniel Gordon Yonatan Bisk Paul G. Allen School of Computer Science and Engineering {jdtho,xkcd,ybisk}@cs.washington.edu Abstract We demonstrate the surprising strength of uni- modal baselines in multimodal domains, and make concrete recommendations for best prac- tices in future research. Where existing work often compares against random or majority class baselines, we argue that unimodal ap- proaches better capture and reflect dataset bi- ases and therefore provide an important com- parison when assessing the performance of multimodal techniques. We present unimodal ablations on three recent datasets in visual nav- igation and QA, seeing an up to 29% absolute gain in performance over published baselines. 1 Introduction All datasets have biases. Baselines should cap- ture these regularities so that outperforming them indicates a model is actually solving a task. In multimodal domains, bias can occur in any sub- set of the modalities. To address this, we argue it is not sufficient for researchers to provide ran- dom or majority class baselines; instead we rec- ommend presenting results for unimodal models. We investigate visual navigation and question an- swering tasks, where agents move through simu- lated environments using egocentric (first person) vision. We find that unimodal ablations (e.g., lan- guage only) in these seemingly multimodal tasks can outperform corresponding full models (§4.1). This work extends observations made in both the Computer Vision (Goyal et al., 2018; Cirik et al., 2018) and Natural Language (Mudrakarta et al., 2018; Glockner et al., 2018; Poliak et al., 2018; Gururangan et al., 2018; Kaushik and Lip- ton, 2018) communities that complex models of- ten perform well by fitting to simple, unintended correlations in the data, bypassing the complex grounding and reasoning that experimenters hoped was necessary for their tasks. F L R U D E F L R U D E F L R U D E F L R U D E t1 t2 t3 t4 Actions: Forward, turn Left & Right, tilt Up & Down, End Figure 1: Navigating without vision leads to sensible navigation trajectories in response to commands like “walk past the bar and turn right”. At t3, “forward” is unavailable as the agent would collide with the wall. We ablate models from three recent papers: (1) navigation (Figure 1) using images of real homes paired with crowdsourced language de- scriptions (Anderson et al., 2018); and (2, 3) navi- gation and egocentric question answering (Gordon et al., 2018; Das et al., 2018a) in simulation with synthetic questions. We find that unimodal abla- tions often outperform the baselines that accom- pany these tasks. Recommendation for Best Practices: Our find- ings show that in the new space of visual nav- igation and egocentric QA, all modalities, even an agent’s action history, are strongly informative. Therefore, while many papers ablate either lan- guage or vision, new results should ablate both. Such baselines expose possible …

Content truncated to 500 words