Robust and Interpretable Grounding of Spatial References with Relation Networks

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1908–1923 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 1908 Robust and Interpretable Grounding of Spatial References with Relation Networks Tsung-Yen Yang1, Andrew S. Lan2, Karthik Narasimhan1 1Princeton University, Princeton, NJ 2 University of Massachusetts, Amherst, MA {ty3, karthikn}@princeton.edu, andrewlan@cs.umass.edu Abstract Learning representations of spatial references in natural language is a key challenge in tasks like autonomous navigation and robotic manip- ulation. Recent work has investigated various neural architectures for learning multi-modal representations for spatial concepts. However, the lack of explicit reasoning over entities makes such approaches vulnerable to noise in input text or state observations. In this paper, we develop effective models for understanding spatial references in text that are robust and interpretable, without sacrificing performance. We design a text-conditioned relation network whose parameters are dynamically computed with a cross-modal attention module to cap- ture fine-grained spatial relations between enti- ties. This design choice provides interpretabil- ity of learned intermediate outputs. Experi- ments across three tasks demonstrate that our model achieves superior performance, with a 17% improvement in predicting goal locations and a 15% improvement in robustness com- pared to state-of-the-art systems. 1 1 Introduction Grounding spatial references in text is essential for effective human-machine communication through natural language. Spatial reasoning is ubiquitous in many scenarios such as autonomous naviga- tion (MacMahon et al., 2006; Vogel and Jurafsky, 2010), situated dialog (Skubic et al., 2002) and robotic manipulation (Landsiedel et al., 2017). De- spite tremendous applicability, understanding spa- tial references is a highly challenging task for cur- rent natural language processing (NLP) systems, requiring a solid contextual understanding of lan- guage dependent on other observations from the en- vironment. Figure 1 demonstrates two tasks where the interpretation of the instruction or statement changes completely with the observation provided. 1Code is available at https://sites.google. com/view/robust-relation-net/home. Figure 1: Two different tasks requiring joint spatial reasoning over observation and text – (top) the same in- struction may specify different goal locations, depend- ing on the map (red flags = goals); (bottom) the same statement may be true or false depending on the image. In the first example, the westernmost circle may lie to the left or right of the navigating agent’s starting location (red dots). In the second, the validity of the statement depends on the relative orientation of the relevant objects – the triangle and the pentagon. Some of the earliest work in this field (Her- skovits, 1987; Regier, 1996) investigated the grounding of spatial prepositions (e.g., “above”, “below”) to perceptual processes like visual sig- nals. Such early grounding efforts were limited by computational bottlenecks but several deep neu- ral architectures have been recently proposed that jointly process text and visual input (Janner et al., 2018; Misra et al., 2017; Bisk et al., 2016; Liu et al., 2019; Jain et al., 2019; Gaddy and Klein, 2019; Hristov et al., 2019; Yu et al., 2018). While these approaches have made significant advances in improving the ability of agents at following …

Content truncated to 500 words