Shaping Visual Representations with Language for Few-Shot Classification

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4823–4830 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 4823 Shaping Visual Representations with Language for Few-Shot Classification Jesse Mu1, Percy Liang1, Noah D. Goodman1,2 Departments of 1Computer Science and 2Psychology Stanford University {muj,ngoodman}@stanford.edu, pliang@cs.stanford.edu Abstract By describing the features and abstractions of our world, language is a crucial tool for hu- man learning and a promising source of super- vision for machine learning models. We use language to improve few-shot visual classifica- tion in the underexplored scenario where nat- ural language task descriptions are available during training, but unavailable for novel tasks at test time. Existing models for this setting sample new descriptions at test time and use those to classify images. Instead, we propose language-shaped learning (LSL), an end-to- end model that regularizes visual representa- tions to predict language. LSL is conceptu- ally simpler, more data efficient, and outper- forms baselines in two challenging few-shot domains. 1 Introduction Humans are powerful and efficient learners par- tially due to the ability to learn from language (Chopra et al., 2019; Tomasello, 1999). For in- stance, we can learn about robins not by seeing thousands of examples, but by being told that a robin is a bird with a red belly and brown feath- ers. This language further shapes the way we view the world, constraining our hypotheses for new concepts: given a new bird (e.g. seagulls), even without language we know that features like belly and feather color are relevant (Goodman, 1955). In this paper, we guide visual representation learning with language, studying the setting where no language is available at test time, since rich linguistic supervision is often unavailable for new concepts encountered in the wild. How can one best use language in this setting? One option is to just regularize, training representations to predict language descriptions. Another is to exploit the compositional nature of language directly by using it as a bottleneck in a discrete latent variable model. c Meta (Snell et al., 2017) Support c Query fθ True LSTM- Dec gϕ L3 (Andreas et al., 2018) Support Query fθ LSTM- Enc hη True LSTM- Dec gϕ (sample from gϕ at test) LSL (ours) fθ fθ a red cross is below a square a red cross is below a square Auxiliary training (discard at test) Figure 1: We propose few-shot classification models whose learned representations are constrained to pre- dict natural language task descriptions during training, in contrast to models which explicitly use language as a bottleneck for classification (Andreas et al., 2018). For example, the recent Learning with Latent Lan- guage (L3; Andreas et al., 2018) model does both: during training, language is used to classify im- ages; at test time, with no language, descriptions are sampled from a decoder conditioned on the language-shaped image embeddings. Whether the bottleneck or regularization most benefits models like L3 is unclear. We disentangle these effects and propose language-shaped learn- ing (LSL), an end-to-end model that …

Content truncated to 500 words