A Computational Acquisition Model for Multimodal Word Categorization

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3819 - 3835 July 10-15, 2022 ©2022 Association for Computational Linguistics A Computational Acquisition Model for Multimodal Word Categorization Uri Berger1,2, Gabriel Stanovsky1, Omri Abend1, and Lea Frermann2 1School of Computer Science and Engineering, The Hebrew University of Jerusalem 2School of Computing and Information Systems, University of Melbourne {uri.berger2, gabriel.stanovsky, omri.abend}@mail.huji.ac.il lea.frermann@unimelb.edu.au Abstract Recent advances in self-supervised modeling of text and images open new opportunities for computational models of child language acqui- sition, which is believed to rely heavily on cross-modal signals. However, prior studies have been limited by their reliance on vision models trained on large image datasets anno- tated with a pre-defined set of depicted object categories. This is (a) not faithful to the infor- mation children receive and (b) prohibits the evaluation of such models with respect to cat- egory learning tasks, due to the pre-imposed category structure. We address this gap, and present a cognitively-inspired, multimodal ac- quisition model, trained from image-caption pairs on naturalistic data using cross-modal self-supervision. We show that the model learns word categories and object recognition abilities, and presents trends reminiscent of those reported in the developmental literature. We make our code and trained models public for future reference and use1. 1 Introduction To date, the mechanisms underlying the efficiency with which infants learn to speak and understand natural language remain an open research question. Research suggests that children leverage contex- tual, inter-personal and non-linguistic information. Visual input is a case in point: when spoken to, infants visually perceive their environment, and paired with the input speech, the visual environ- ment could help bootstrap linguistic knowledge (Tomasello et al., 1996). Unlike social cues, visual input has a natural physical representation, in the form of pixel maps or videos. Previous multimodal language acquisition stud- ies either considered toy scenarios with small vo- cabularies (Roy and Pentland, 2002; Frank et al., 2007), or used visual encoders that were pretrained 1github.com/SLAB-NLP/multimodal_clustering Figure 1: Model overview. Given an image-caption pair (left), both the visual (top) and textual encoder (bottom) generate a binary vector indicative of the clus- ters associated with the current input. The text/visual model predicts a probability vector over clusters per word/image. Vectors are mapped to a binary space using thresholding. The modality-specific vectors pro- vide mutual supervision during training (right). on large labeled data bases such as ImageNet (Deng et al., 2009) or Visual Genome (Krishna et al., 2017). This has two drawbacks: first, systematic access to labeled data is a cognitively implausible assumption in a language acquisition setting; sec- ond, imposing a pre-defined categorization system precludes studying categories that emerge when learning from unlabeled multimodal data. This type of setting more closely resembles the data un- derlying early language learning at a time when the child has only acquired little conceptual informa- tion. Although the subject of much psycholinguis- tic work, the computational study of multimodal word categories, formed without recourse to …

Content truncated to 500 words