Learning grounded word meaning representations on similarity graphs

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4760–4769 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 4760 Learning grounded word meaning representations on similarity graphs Mariella Dimiccoli Institut de Robòtica i Informàtica Industrial (CSIC-UPC), Barcelona, Spain mdimiccoli@iri.upc.edu Herwig Wendt CNRS, IRIT Univ. of Toulouse, France herwig.wendt@irit.fr Pau Batlle ∗ California Institute of Technology Pasadena, California Abstract This paper introduces a novel approach to learn visually grounded meaning represen- tations of words as low-dimensional node embeddings on an underlying graph hierar- chy. The lower level of the hierarchy mod- els modality-specific word representations through dedicated but communicating graphs, while the higher level puts these representa- tions together on a single graph to learn a rep- resentation jointly from both modalities. The topology of each graph models similarity re- lations among words, and is estimated jointly with the graph embedding. The assumption underlying this model is that words sharing similar meaning correspond to communities in an underlying similarity graph in a low- dimensional space. We named this model Hi- erarchical Multi-Modal Similarity Graph Em- bedding (HM-SGE). Experimental results vali- date the ability of HM-SGE to simulate human similarity judgements and concept categoriza- tion, outperforming the state of the art. 1 1 Introduction During the last decade, there has been an increasing interest in deriving semantic representations from text corpora in terms of word vector space for both words in context and words in isolation (Mikolov et al., 2013; Ling et al., 2015; McCann et al., 2017; Devlin et al., 2018; Peters et al., 2018). However, semantic representations are tied to sensory ex- perience, at least for concrete nouns (Anderson et al., 2017), and deriving them by relying solely on text leads to lack of grounding on the extra- language modality. For this reason, several works have addressed the problem of grounding percep- tual information in the form of visual informa- tion approximated by feature norms elicited from ∗ Work done during an internship at the IRI (CSIC-UPC). 1Code available: https://github.com/mdimiccoli/HM-SGE/. Work partially funded by projects MINECO/ERDF RyC, PID2019-110977GA-I00, MDM-2016-0656. humans (Andrews et al., 2009; Silberer and Lap- ata, 2012), or extracted automatically from images (Kiela and Bottou, 2014; Lazaridou et al., 2015; Sil- berer et al., 2016), or a combination of them (Roller and Im Walde, 2013). Furthermore, several inte- gration mechanisms for the linguistic and percep- tual modalities have been proposed. They include methods employing transformation and dimension reduction on the concatenation of unimodal repre- sentations (Bruni et al., 2014; Hill and Korhonen, 2014); generative probabilistic models (Andrews et al., 2009; Roller and Im Walde, 2013; Feng and Lapata, 2010); deep learning methods such as au- toencoders and recursive neural networks (Silberer et al., 2016; Socher et al., 2013). While the ap- proaches above take as input previously extracted unimodal representations, (Lazaridou et al., 2015; Hill and Korhonen, 2014; Zablocki et al., 2018) learn directly from raw data by integrating both modalities non-linearly in a low-dimensional space within a skip-gram model framework. These ap- proaches …

Content truncated to 500 words