Learning Representations Specialized in Spatial Knowledge: Leveraging Language and Vision

Learning Representations Specialized in Spatial Knowledge: Leveraging Language and Vision Guillem Collell Department of Computer Science KU Leuven 3001 Heverlee, Belgium gcollell@kuleuven.be Marie-Francine Moens Department of Computer Science KU Leuven 3001 Heverlee, Belgium sien.moens@cs.kuleuven.be Abstract Spatial understanding is crucial in many real- world problems, yet little progress has been made towards building representations that capture spatial knowledge. Here, we move one step forward in this direction and learn such representations by leveraging a task consisting in predicting continuous 2D spa- tial arrangements of objects given object- relationship-object instances (e.g., “cat under chair”) and a simple neural network model that learns the task from annotated images. We show that the model succeeds in this task and, furthermore, that it is capable of predicting correct spatial arrangements for unseen ob- jects if either CNN features or word embed- dings of the objects are provided. The differ- ences between visual and linguistic features are discussed. Next, to evaluate the spatial representations learned in the previous task, we introduce a task and a dataset consisting in a set of crowdsourced human ratings of spatial similarity for object pairs. We find that both CNN (convolutional neural network) features and word embeddings predict human judgments of similarity well and that these vectors can be further specialized in spatial knowledge if we update them when training the model that predicts spatial arrangements of objects. Overall, this paper paves the way towards building distributed spatial represen- tations, contributing to the understanding of spatial expressions in language. 1 Introduction Representing spatial knowledge is instrumental in any task involving text-to-scene conversion such as robot understanding of natural language commands (Guadarrama et al., 2013; Moratz and Tenbrink, 2006) or a number of robot navigation tasks. Despite recent advances in building specialized representa- tions in domains such as sentiment analysis (Tang et al., 2014), semantic similarity/relatedness (Kiela et al., 2015) or dependency parsing (Bansal et al., 2014), little progress has been made towards build- ing distributed representations (a.k.a. embeddings) specialized in spatial knowledge. Intuitively, one may reasonably expect that the more attributes two objects share (e.g., size, func- tionality, etc.), the more likely they are to exhibit similar spatial arrangements with respect to other objects. Leveraging this intuition, we foresee that visual and linguistic representations can be spatially informative about unseen objects as they encode features/attributes of objects (Collell and Moens, 2016). For instance, without having ever seen an “elephant” before, but only a “horse”, one would probably devise the “elephant” carrying the “hu- man” than otherwise, just by considering their size attribute. Similarly, one can infer that a “tablet” and a “book” will show similar spatial patterns (usually on a table, in someone’s hands, etc.) although they barely show any visual resemblance—yet they are similar in size and functionality. In this paper we systematically study how informative visual and lin- guistic features—in the form of convolutional neural network (CNN) features and word embeddings—are about the spatial behavior of objects. An important goal of this work is to learn dis- tributed representations …

Content truncated to 500 words