Vision and Feature Norms: Improving automatic feature norm learning through cross-modal maps

Proceedings of NAACL-HLT 2016, pages 579–588, San Diego, California, June 12-17, 2016. c⃝2016 Association for Computational Linguistics Vision and Feature Norms: Improving automatic feature norm learning through cross-modal maps Luana Bulat, Douwe Kiela and Stephen Clark Computer Laboratory University of Cambridge ltf24,douwe.kiela,stephen.clark@cl.cam.ac.uk Abstract Property norms have the potential to aid a wide range of semantic tasks, provided that they can be obtained for large numbers of concepts. Recent work has focused on text as the main source of information for auto- matic property extraction. In this paper we ex- amine property norm prediction from visual, rather than textual, data, using cross-modal maps learnt between property norm and visual spaces. We also investigate the importance of having a complete feature norm dataset, for both training and testing. Finally, we evalu- ate how these datasets and cross-modal maps can be used in an image retrieval task. 1 Introduction Many cognitive theories of conceptual organisation assume that concepts are distributed representations over semantic primitives, often referred to as fea- tures or properties1 (Tyler et al., 2000; Randall et al., 2004). That is, we can understand the meaning of a concept through its properties. For example, under- standing the meaning of BANANA is closely related to understanding that it has properties such as is a fruit, is yellow, is long, is sweet, and knowing how these properties overlap with or differ from the prop- erties of other concepts. A number of property norm datasets, where hu- mans were asked to list attributes of given concepts, have been collected to test this hypothesis (McRae et al., 2005; Vinson and Vigliocco, 2008; Devereux 1Throughout the paper we will be using the terms properties, features norms and attributes interchangeably. et al., 2013). After having been used to test mod- els of conceptual representation in cognitive science for decades (Randall et al., 2004; Cree et al., 2006), these datasets have proved to be useful in a wide range of semantic NLP tasks as well, including text simplification for limited vocabulary groups. More recently, property norms have been used as a proxy for perceptual information in a number of studies on multi-modal semantics (Andrews et al., 2009; Ri- ordan and Jones, 2011; Silberer and Lapata, 2012; Roller and Im Walde, 2013; Hill and Korhonen, 2014). Such models aim to addres the grounding problem (Harnad, 1990) that distributional semantic models of language (Turney and Pantel, 2010; Clark, 2015) suffer from. Property norms are a valuable source of seman- tic information, and can potentially be applied to a variety of NLP tasks, but are expensive to obtain because they involve intensive human annotation. The largest property norm dataset to date consists of just 638 concepts (Devereux et al., 2013), and the most widely cited one presents properties for only 541 concepts (McRae et al., 2005). If we are to use these datasets in large-scale semantic tasks, we would need to extend the currently available prop- erty norms by obtaining annotations for more than just a few hundred words. The alternative to …

Content truncated to 500 words