Language-Mediated, Object-Centric Representation Learning

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2033–2046 August 1–6, 2021. ©2021 Association for Computational Linguistics 2033 Language-Mediated, Object-Centric Representation Learning Ruocheng Wang∗ Stanford University Jiayuan Mao∗ MIT CSAIL Samuel J. Gershman† Harvard University Jiajun Wu† Stanford University Abstract We present Language-mediated, Object- centric Representation Learning (LORL), a paradigm for learning disentangled, object- centric scene representations from vision and language. LORL builds upon recent advances in unsupervised object discovery and segmen- tation, notably MONet and Slot Attention. While these algorithms learn an object-centric representation just by reconstructing the input image, LORL enables them to further learn to associate the learned representations to concepts, i.e., words for object categories, properties, and spatial relationships, from language input. These object-centric concepts derived from language facilitate the learning of object-centric representations. LORL can be integrated with various unsuper- vised object discovery algorithms that are language-agnostic. Experiments show that the integration of LORL consistently improves the performance of unsupervised object discovery methods on two datasets via the help of language. We also show that concepts learned by LORL, in conjunction with object discovery methods, aid downstream tasks such as referring expression comprehension. 1 Introduction Cognitive studies show that human infants develop object individuation skill from diverse sources of information: spatial-temporal information, object property information, and language (Xu, 1999, 2007; Westermann and Mareschal, 2014). Specifi- cally, young infants develop object-based attention that disentangles the motion and location of objects from their visual appearance features. Later on, they can leverage the knowledge acquired through word learning to solve the problem of object indi- viduation: words provide clues about object iden- tity and type. The general picture from cognitive science is that object perception and language co- develop in support of one another (Bloom, 2002). ∗, †: indicates equal contribution Input Image Segmentation I Segmentation II Q: What is the black object in the image? A: Plate. ☓ Q: What is the black object in the image? A: Pan. ✓ Input Image Segmentation I Segmentation II Q: How many legs are visible? A: 3. ☓ Q: How many legs are visible? A: 4. ✓ (b) (a) (PartNet) (ShopVRB) Figure 1: Two illustrative cases of Language-mediated, Object-centric Representation Learning. Different col- ors in the segmentation masks indicate individual ob- jects recognized by the model. LORL can learn from vi- sual and language inputs to associate various concepts: black, pan, leg, with the visual appearance of individ- ual objects. Furthermore, language provides cues about how an input scene should be segmented into individ- ual objects: (a) segmenting the frying pan and its han- dle into two parts (Segmentation II) yields an incorrect answer to the question ; (b) an incorrect parsing of the chair image makes the counting result wrong. Our long-term goal is to endow machines with similar abilities. In this paper, we focus on how language may support object discovery and seg- mentation. Recent work has studied the prob- lem of unsupervised object representation learning, though without language. As an example, factor- ized, object-centric scene …

Content truncated to 500 words