A Visuospatial Dataset for Naturalistic Verb Learning

Proceedings of the Ninth Joint Conference on Lexical and Computational Semantics (*SEM), pages 143–153 Barcelona, Spain (Online), December 12–13, 2020 143 A Visuospatial Dataset for Naturalistic Verb Learning Dylan Ebert Brown University dylan ebert@brown.edu Ellie Pavlick Brown University ellie pavlick@brown.edu Abstract We introduce a new dataset for training and evaluating grounded language models. Our data is collected within a virtual reality envi- ronment and is designed to emulate the qual- ity of language data to which a pre-verbal child is likely to have access: That is, natu- ralistic, spontaneous speech paired with richly grounded visuospatial context. We use the col- lected data to compare several distributional semantics models for verb learning. We evalu- ate neural models based on 2D (pixel) features as well as feature-engineered models based on 3D (symbolic, spatial) features, and show that neither modeling approach achieves satisfac- tory performance. Our results are consistent with evidence from child language acquisition that emphasizes the difficulty of learning verbs from naive distributional data. We discuss av- enues for future work on cognitively-inspired grounded language learning, and release our corpus with the intent of facilitating research on the topic. 1 Introduction While distributional models of semantics have seen incredible success in recent years (Devlin et al., 2018), most current models lack “ground- ing”, or a connection between words and their referents in the non-linguistic world. Ground- ing is an important aspect to representations of meaning and arguably lies at the core of lan- guage “understanding” (Bender and Koller, 2020). Work on grounded language learning has tended to make opportunistic use of large available cor- pora, e.g. by learning from web-scale corpora of image (Bruni et al., 2012) or video captions (Sun et al., 2019), or has been driven by particular downstream applications such as robot navigation (Anderson et al., 2018). In this work, we take an aspirational look at grounded distributional semantics models, based on the type of situated contexts and weak supervi- sion from which children are able to learn much of their early vocabulary. Our approach is mo- tivated by the assumption that building computa- tional models which emulate human language pro- cessing is in itself a worthwhile endeavor, which can yield both scientific (Potts, 2019) and engi- neering (Linzen, 2020) advances in NLP. Thus, we aim to develop a dataset that better reflects both the advantages and the challenges of humans’ naturalistic learning environments. For example, unlike most vision-and-language models, children likely have the advantage of access to symbolic representations of objects and their physics prior to beginning word learning (Spelke and Kinzler, 2007). However, also unlike NLP models, which are typically trained on image or video captions with strong signal, children’s language input is highly unstructured and the content is often hard to predict given only the grounded context (Gillette et al., 1999). We make two main contributions. First (§2), using a virtual reality kitchen environment, we collect and release1 the New Brown Corpus2: A dataset containing 18K words of spontaneous speech alongside rich visual and spatial …

Content truncated to 500 words