Deriving continous grounded meaning representations from referentially structured multimodal contexts

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 959–965 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Deriving continous grounded meaning representations from referentially structured multimodal contexts Sina Zarrieß and David Schlangen Dialogue Systems Group // CITEC // Faculty of Linguistics and Literary Studies Bielefeld University, Germany {sina.zarriess,david.schlangen}@uni-bielefeld.de Abstract Corpora of referring expressions paired with their visual referents are a good source for learning word meanings di- rectly grounded in visual representations. Here, we explore additional ways of ex- tracting from them word representations linked to multi-modal context: through expressions that refer to the same ob- ject, and through expressions that refer to different objects in the same scene. We show that continuous meaning rep- resentations derived from these contexts capture complementary aspects of sim- ilarity, even if not outperforming tex- tual embeddings trained on very large amounts of raw text when tested on stan- dard similarity benchmarks. We propose a new task for evaluating grounded meaning representations—detection of potentially co-referential phrases—and show that it requires precise denotational representa- tions of attribute meanings, which our method provides. 1 Introduction Various routes for linking language to extra- linguistic context have been explored in recent years. A lot of research has looked at integrating visual representations, either directly (Matuszek et al., 2012; Krishnamurthy and Kollar, 2013; Yu et al., 2016; Schlangen et al., 2016) or through mapping into a multi-modal distributional space (Feng and Lapata, 2010; Bruni et al., 2012; Kiela and Bottou, 2014; Lazaridou et al., 2015). Young et al. (2014) have explored a less direct link, by representing the extension of phrasal expressions as sets of images, and deriving from this a pre- cise notion of denotational similarity. In very re- cake girl young lady grandma blue shirt table old lady denotational similarity visual similarity situational similarity Figure 1: Dimensions of context in referential, visually grounded language, and similarity relations that can be de- rived from it, image from MSCOCO (Lin et al., 2014)) cent work, Cocos and Callison-Burch (2017) use spatial context from geo-located tweets to induce word embeddings that capture situational similar- ity between lexical items. In this paper, we explore an approach that com- bines aspects of several of these paths. Start- ing point is the observation that corpora of ex- ophoric referring expressions provide richly struc- tured contexts that go beyond just linking indi- vidual expressions with their denotations. As an example consider the scene in Figure 1 depicting several referents and corresponding referring ex- pressions produced by different speakers. This scene provides a learner not only with an exam- ple of a referent for the word lady, it also pro- vides the information that lady can co-refer with girl, and that its denotations can spatially / situ- ationally co-occur with those of table and cake. From these types of information we infer word embeddings, following the method from Levy and Goldberg (2014) for training embeddings on arbi- 959 trary non-linear context, and we show that these capture complementary …

Content truncated to 500 words