Cross-View Language Modeling: Towards Unified Cross-Lingual Cross-Modal Pre-training

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 524–544 July 9-14, 2023 ©2023 Association for Computational Linguistics World-to-Words: Grounded Open Vocabulary Acquisition through Fast Mapping in Vision-Language Models Ziqiao Ma∗ Jiayi Pan∗ Joyce Chai Computer Science and Engineering Division, University of Michigan {marstin,jiayipan,chaijy}@umich.edu Abstract The ability to connect language units to their referents in the physical world, referred to as grounding, is crucial to learning and under- standing grounded meanings of words. While humans demonstrate fast mapping in new word learning, it remains unclear whether modern vision-language models can truly represent lan- guage with their grounded meanings, and how grounding may further bootstrap new word learning. To this end, we introduce Grounded Open Vocabulary Acquisition (GOVA) to ex- amine grounding and bootstrapping in open- world language learning. As an initial at- tempt, we propose World-to-Words (W2W), a novel visually-grounded language model by pre-training on image-text pairs highlighting grounding as an objective. Through extensive experiments and analysis, we demonstrate that W2W is a more coherent and fast grounded word learner, and that the grounding ability acquired during pre-training helps the model to learn unseen words more rapidly and robustly.1 1 Introduction Language is learned through sensorimotor experi- ence in the physical world (Bisk et al., 2020). The ability to connect language units to their referents in the physical world, referred to as grounding, plays an important role in learning and understand- ing grounded meanings of words (Harnad, 1990). As shown in Figure 1, a human reader would eas- ily ground noun phrases to the corresponding en- tities captured in the image. Even when the term “incinerator” is new to human learners, they can still locate the object of interest through the lan- guage and visual context, and acquire its meaning. In fact, this ability to bootstrap new word learn- ing with only minimal information, known as fast mapping, is demonstrated abundantly in cognitive ∗Equal contribution. 1Code available at https://github.com/sled-group/ world-to-words. A lady wearing a navy blue stripe tank top is getting ready to burn glass in front of an incinerator. Figure 1: Even when the term “incinerator” (high- lighted yellow) is new to human learners, they can still locate the most likely referent (indicated by the yellow bounding box) in the perceived world by grounding. literature on human language acquisition (Carey and Bartlett, 1978; Carey, 1978; Golinkoff et al., 2000; Smith and Yu, 2008). Recently, there has been a substantial effort on pre-training vision-language models (VLMs) (Du et al., 2022a). Despite the exciting performance of these models on a variety of downstream vision and language tasks, it remains unclear whether these models can truly understand or produce language with their grounded meanings in the perceived world, and how grounding may further bootstrap new word learning. These questions are of inter- est from both a scientific and an engineering point of view. From a scientific perspective, ground- ing is crucial to language learners, as children at- tend to intended objects in the environment …

Content truncated to 500 words