Decoding Language Spatial Relations to 2D Spatial Arrangements

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4549–4560 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 4549 Decoding Language Spatial Relations to 2D Spatial Arrangements Gorjan Radevski1, Guillem Collell2, Marie-Francine Moens2, Tinne Tuytelaars1 1Department of Electrical Engineering (ESAT) 2Department of Computer Science (CS) KU Leuven {gorjan.radevski, tinne.tuytelaars}@esat.kuleuven.be {guillem.collell, sien.moens}@cs.kuleuven.be Abstract We address the problem of multimodal spatial understanding by decoding a set of language- expressed spatial relations to a set of 2D spa- tial arrangements in a multi-object and multi- relationship setting. We frame the task as ar- ranging a scene of clip-arts given a textual description. We propose a simple and effec- tive model architecture SPATIAL-REASONING BERT (SR-BERT), trained to decode text to 2D spatial arrangements in a non-autoregressive manner. SR-BERT can decode both explicit and implicit language to 2D spatial arrange- ments, generalizes to out-of-sample data to a reasonable extent and can generate complete abstract scenes if paired with a clip-arts pre- dictor. Finally, we qualitatively evaluate our method with a user study, validating that our generated spatial arrangements align with hu- man expectation. 1 Introduction Spatial understanding is a problem of paramount importance to both the vision and the language community. For a machine learning model to be able to reason about the spatial domain w.r.t. an- other modality (language, vision, etc.), it should in- corporate common-sense spatial knowledge, which is often obvious to humans, yet hard to grasp by machines. If the spatial relations are expressed in language, such common sense knowledge can be hidden within, e.g., “mike and jenny see a duck” (Figure 1) – meaning that both “Mike” and “Jenny” should be facing the “duck”. The complexity of the problem increases tremendously when there is no limit on the number/type of objects and rela- tionships – the de-facto setting in computer graph- ics, video games, 3D modelling, etc. Communi- cating spatial relations via language is intuitive for humans, while arranging objects in a multi- dimensional space is tedious. Therefore, building jenn� ha� a hamb��ge�. mike i� ��anding �nde� a helico��e�. mike and jenn� a�e ha���. mike and jenn� a�e ��anding ne�� �o �he �able. �he�e i� a helico��e�. jenn� i� holding a hamb��ge� . Gro�nd �r��h Genera�ed a d�ck i� �ea�ing a helm. mike i� la�ghing a� �he d�ck. mike and jenn� a�e ha��� �o �ee a d�ck. �he d�ck i� �ea�ing a� ha�. a d�ck ��and� be��een mike and jenn�. jenn� i� ��anding �nde� �he ��ee. Figure 1: Given a set of clip-arts and a textual descrip- tion of a scene, including both implicit as well as ex- plicit language, our method automatically generates a reasonable spatial arrangement. models that exhibit spatial understanding is a key step towards automation – aiding humans in the repetitive time-consuming tasks. To date, multiple methods that explicitly investi- gate spatial reasoning in a multidimensional space have been proposed. However, the main limita- tions are: (i) scene environments with strong priors on (relative) object placements (e.g., indoor home environments); (Chang et al., 2017; …

Content truncated to 500 words