Learning Functional Distributional Semantics with Visual Data

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 3976 - 3988 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Learning Functional Distributional Semantics with Visual Data Yinhong Liu, Guy Emerson Department of Computer Science and Technology University of Cambridge {yl535, gete2}@cam.ac.uk Abstract Functional Distributional Semantics is a re- cently proposed framework for learning dis- tributional semantics that provides linguistic interpretability. It models the meaning of a word as a binary classifier rather than a numeri- cal vector. In this work, we propose a method to train a Functional Distributional Semantics model with grounded visual data. We train it on the Visual Genome dataset, which is closer to the kind of data encountered in human lan- guage acquisition than a large text corpus. On four external evaluation datasets, our model out- performs previous work on learning semantics from Visual Genome. 1 1 Introduction The target of distributional semantics models is to understand and represent the meanings of words from their distributions in large corpus. Many approaches learn a numerical vector for each word, which encodes its distributional informa- tion. They can be roughly divided into two cat- egories: frequency-based methods such as co- occurrence matrix (Sahlgren, 2006), and prediction- based methods such as Word2vec (Mikolov et al., 2013). More recently, progress has been made in learning word representations in a specific context, which are also called contextualized embeddings. Examples include ELMo (Peters et al., 2018) and BERT (Devlin et al., 2019). Functional Distributional Semantics is a frame- work that not only provides contextualized seman- tic representations, but also provides more inter- pretability. It was first proposed by Emerson and Copestake (2016), and it explicitly separates the modeling of words and the modeling of objects and events. This is a fundamental distinction in predi- cate logic. While logic is not necessary for all NLP 1Our code and models are publicly available at: https: //github.com/williamLyh/PixieVGModel tasks, it is an essential tool for modeling many se- mantic phenomena (for example, see: Cann, 1993; Allan, 2001; Kamp and Reyle, 2013). For semantic research questions, having a logical interpretation is a clear advantage over vector-based models. We will explain the framework in Section 2.2. Another issue with distributional semantic mod- els, as discussed by Emerson (2020c), is the sym- bol grounding problem – if meanings of words are defined in terms of other words, the definitions are circular. During human language acquisition, words are learned while interacting with the phys- ical world, rather than from text or speech alone. An important goal for a semantic theory is to ex- plain how language relates to the world, and how this relationship is learned. We focus on the Visual Genome dataset, not only because it provides rela- tively fine-grained annotations, but also it is similar to realistic circumstance encountered during lan- guage acquisition, as we will explain in Section 2.3. Our main theoretical contribution is to adapt the Functional Distributional Semantics framework to better suit visual data. This is …

Content truncated to 500 words