Multi-Modal Models for Concrete and Abstract Concept Meaning

Multi-Modal Models for Concrete and Abstract Concept Meaning Felix Hill Computer Laboratory University of Cambridge fh295@cam.ac.uk Roi Reichart Technion - IIT Haifa, Israel roiri@ie.technion.ac.il Anna Korhonen Computer Laboratory University of Cambridge alk23@cam.ac.uk Abstract Multi-modal models that learn semantic rep- resentations from both linguistic and percep- tual input outperform language-only models on a range of evaluations, and better reflect human concept acquisition. Most perceptual input to such models corresponds to concrete noun concepts and the superiority of the multi- modal approach has only been established when evaluating on such concepts. We there- fore investigate which concepts can be effec- tively learned by multi-modal models. We show that concreteness determines both which linguistic features are most informative and the impact of perceptual input in such mod- els. We then introduce ridge regression as a means of propagating perceptual informa- tion from concrete nouns to more abstract con- cepts that is more robust than previous ap- proaches. Finally, we present weighted gram matrix combination, a means of combining representations from distinct modalities that outperforms alternatives when both modalities are sufficiently rich. 1 Introduction What information is needed to learn the meaning of a word? Children learning words are exposed to a diverse mix of information sources. These include clues in the language itself, such as nearby words or speaker intention, but also what the child perceives about the world around it when the word is heard. Learning the meaning of words requires not only a sensitivity to both linguistic and perceptual input, but also the ability to process and combine informa- tion from these modalities in a productive way. Many computational semantic models represent words as real-valued vectors, encoding their rela- tive frequency of occurrence in particular forms and contexts in linguistic corpora (Sahlgren, 2006; Tur- ney et al., 2010). Motivated both by parallels with human language acquisition and by evidence that many word meanings are grounded in the percep- tual system (Barsalou et al., 2003), recent research has explored the integration into text-based models of input that approximates the visual or other sen- sory modalities (Silberer and Lapata, 2012; Bruni et al., 2014). Such models can learn higher-quality semantic representations than conventional corpus- only models, as evidenced by a range of evaluations. However, the majority of perceptual input for the models in these studies corresponds directly to con- crete noun concepts, such as chocolate or cheese- burger, and the superiority of the multi-modal over the corpus-only approach has only been established when evaluations include such concepts (Leong and Mihalcea, 2011; Bruni et al., 2012; Roller and Schulte im Walde, 2013; Silberer and Lapata, 2012). It is thus unclear if the multi-modal approach is ef- fective for more abstract words, such as guilt or obe- sity. Indeed, since empirical evidence indicates dif- ferences in the representational frameworks of both concrete and abstract concepts (Paivio, 1991; Hill et al., 2013), and verb and noun concepts (Markman and Wisniewski, 1997), perceptual information may not fulfill the same role in the representation of the various concept …

Content truncated to 500 words