A Multimodal LDA Model integrating Textual, Cognitive and Visual Modalities

Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1146–1157, Seattle, Washington, USA, 18-21 October 2013. c⃝2013 Association for Computational Linguistics A Multimodal LDA Model Integrating Textual, Cognitive and Visual Modalities Stephen Roller Department of Computer Science The University of Texas at Austin roller@cs.utexas.edu Sabine Schulte im Walde Institut f¨ur Maschinelle Sprachverarbeitung Universit¨at Stuttgart schulte@ims.uni-stuttgart.de Abstract Recent investigations into grounded models of language have shown that holistic views of language and perception can provide higher performance than independent views. In this work, we improve a two-dimensional multi- modal version of Latent Dirichlet Allocation (Andrews et al., 2009) in various ways. (1) We outperform text-only models in two different evaluations, and demonstrate that low-level visual features are directly compatible with the existing model. (2) We present a novel way to integrate visual features into the LDA model using unsupervised clusters of images. The clusters are directly interpretable and im- prove on our evaluation tasks. (3) We provide two novel ways to extend the bimodal mod- els to support three or more modalities. We find that the three-, four-, and five-dimensional models significantly outperform models using only one or two modalities, and that nontex- tual modalities each provide separate, disjoint knowledge that cannot be forced into a shared, latent structure. 1 Introduction In recent years, an increasing body of work has been devoted to multimodal or “grounded” models of lan- guage where semantic representations of words are extended to include perceptual information. The un- derlying hypothesis is that the meanings of words are explicitly tied to our perception and understand- ing of the world around us, and textual-information alone is insufficient for a complete understanding of language. The language grounding problem has come in many different flavors with just as many different ap- proaches. Some approaches apply semantic parsing, where words and sentences are mapped to logical structure meaning (Kate and Mooney, 2007). Oth- ers provide automatic mappings of natural language instructions to executable actions, such as interpret- ing navigation directions (Chen and Mooney, 2011) or robot commands (Tellex et al., 2011; Matuszek et al., 2012). Some efforts have tackled tasks such as automatic image caption generation (Feng and La- pata, 2010a; Ordonez et al., 2011), text illustration (Joshi et al., 2006), or automatic location identifica- tion of Twitter users (Eisenstein et al., 2010; Wing and Baldridge, 2011; Roller et al., 2012). Another line of research approaches grounded language knowledge by augmenting distributional approaches of word meaning with perceptual infor- mation (Andrews et al., 2009; Steyvers, 2010; Feng and Lapata, 2010b; Bruni et al., 2011; Silberer and Lapata, 2012; Johns and Jones, 2012; Bruni et al., 2012a; Bruni et al., 2012b; Silberer et al., 2013). Although these approaches have differed in model definition, the general goal in this line of research has been to enhance word meaning with perceptual information in order to address one of the most com- mon criticisms of distributional semantics: that the “meaning of words is entirely given by other words” (Bruni et al., …

Content truncated to 500 words