Understanding Spatial Relations through Multiple Modalities

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 2368–2372 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 2368 Understanding Spatial Relations through Multiple Modalities Soham Dan, Hangfeng He, Dan Roth University of Pennsylvania {sohamdan, hangfeng, danroth}@seas.upenn.edu Abstract Recognizing spatial relations and reasoning about them is essential in multiple applications including navigation, direction giving and human-computer interaction in general. Spatial relations between objects can either be explicit – expressed as spatial prepositions, or implicit – expressed by spatial verbs such as moving, walking, shifting, etc. Both these, but implicit relations in particular, require significant common sense understanding. In this paper, we introduce the task of inferring implicit and explicit spatial relations between two entities in an image. We design a model that uses both textual and visual information to predict the spatial relations, making use of both positional and size information of objects and image embeddings. We contrast our spatial model with powerful language models and show how our modeling complements the power of these, improving prediction accuracy and coverage and facilitates dealing with unseen subjects, objects and relations. Keywords: Knowledge Discovery/Representation Figure 1: woman [?] bed. Language models adopt the choice seen most commonly, i.e., sleeping on, but we pro- pose an image-specific model. 1. Introduction Humans are able to do common sense reasoning across a variety of modalities – textual, visual and for a vari- ety of tasks – reasoning, locating, navigation. Several such tasks require spatial knowledge understanding and reasoning (Kordjamshidi et al., 2010), (Kordjamshidi et al., 2011), (Johnson et al., 2017), (Wang et al., 2017), (Baldridge et al., 2018). Although, there has been sev- eral recent works on common-sense reasoning (Speer and Havasi, 2012), (Emami et al., 2018), (Sap et al., 2018), (Storks et al., 2019), progress on spatial understanding and common-sense is rather limited. Prior work has either not been spatially focused (Lin and Parikh, 2016), (Xu et al., 2015),(Antol et al., 2015) or very restrictive in the class of spatial relations they handle (Xu et al., 2017), (Kord- jamshidi et al., 2017). Recently, (Collell et al., 2018) pre- sented the task of predicting an object’s location and size in an image given the subject’s bounding box and the spatial relation between them. In this paper, we address the problem of understanding spatial relations. We specifically want to infer the spatial relationship between two entities given an image involv- ing them. Spatial relations can either be – explicit (spatial prepositions such as on, above, under) or implicit (intrinsic spatial concepts associated with actions – sleeping, sitting, flying). We take as input an image and the bounding boxes of the entities which are spatially related, the word and im- age embeddings of the two entities, and we want to pre- dict the spatial relation between them (eg: sleeping, stand- ing, sitting-on in Figure 1. We compare this with power- ful language models like BERT (Devlin et al., 2018) which have been trained on very large text corpora …

Content truncated to 500 words