Encoding Lexico-Semantic Knowledge using Ensembles of Feature Maps from Deep Convolutional Neural Networks

Proceedings of the 28th International Conference on Computational Linguistics, pages 1906–1921 Barcelona, Spain (Online), December 8-13, 2020 1906 Encoding Lexico-Semantic Knowledge using Ensembles of Feature Maps from Deep Convolutional Neural Networks Steven Derby Paul Miller Barry Devereux Queen’s University Belfast, Belfast, United Kingdom {sderby02, p.miller, b.devereux}@qub.ac.uk Abstract Semantic models derived from visual information have helped to overcome some of the limita- tions of solely text-based distributional semantic models. Researchers have demonstrated that text and image-based representations encode complementary semantic information, which when combined provide a more complete representation of word meaning, in particular when com- pared with data on human conceptual knowledge. In this work, we reveal that these vision-based representations, whilst quite effective, do not make use of all the semantic information available in the neural network that could be used to inform vector-based models of semantic representa- tion. Instead, we build image-based meta-embeddings from computer vision models, which can incorporate information from all layers of the network, and show that they encode a richer set of semantic attributes and yield a more complete representation of human conceptual knowledge. 1 Introduction Many approaches to representing the meaning of imageable, concrete concepts (e.g. FROG, APPLE, CAR, GUITAR) have been developed in the fields of cognitive science, computational linguistics and computer vision. Most explicitly, property listing studies have been used in cognitive psychology and cognitive neuroscience to characterise word meaning in terms of discrete semantic properties (McRae et al., 2005; Devereux et al., 2014; Buchanan et al., 2019). In property listing studies, human participants enumerate as many features as they can for each concept word, and these responses are then aggregated and normalised to a set of verbal semantic descriptors that correspond to elements of concept meaning (e.g. does-croak for FROG). This gives a representation of each concept as a sparse vector which encodes the semantic properties that occur for that concept. The resulting properties can then been applied to research investigating the organisation of semantic processing across the cortex, and to studies of the speed or ease of semantic processing for different concepts and different types of concept knowledge (Fieder et al., 2019; Evans et al., 2019; Kivisaari et al., 2019a; Bruffaerts et al., 2019). A desirable trait of semantic property norms is their interpretability, since this interpretability facili- tates the design of cognitive experiments on conceptual semantics (Murphy, 2004). This interpretability has also allowed researchers in NLP interested in distributional lexical semantics to gain better insights into the kinds of information that dense vector space models attain from pretraining. Even though many state-of-the-art vector space models of word meaning perform well when evaluated on both intrinsic and downstream tasks (Mikolov et al., 2013; Pennington et al., 2014; Bojanowski et al., 2017), researchers have demonstrated that such models often fail to fully encode certain facets of conceptual meaning (Li and Gauthier, 2017). For example, taxonomic properties (i.e. properties describing the object category, such as is-an-amphibian for FROG) and properties reflecting encyclopedic information or information about object function tend to be …

Content truncated to 500 words