SPARTQA: A Textual Question Answering Benchmark for Spatial Reasoning

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4582–4598 June 6–11, 2021. ©2021 Association for Computational Linguistics 4582 SPARTQA: A Textual Question Answering Benchmark for Spatial Reasoning Roshanak Mirzaee⋆ Hossein Rajaby Faghihi⋆ Qiang Ning♠∗Parisa Kordjamshidi⋆ ⋆Michigan State University ♠Amazon {mirzaeem,rajabyfa,kordjams}@msu.edu qning@amazon.com Abstract This paper proposes a question-answering (QA) benchmark for spatial reasoning on nat- ural language text which contains more real- istic spatial phenomena not covered by prior work and is challenging for state-of-the-art language models (LM). We propose a distant supervision method to improve on this task. Specifically, we design grammar and reason- ing rules to automatically generate a spatial de- scription of visual scenes and corresponding QA pairs. Experiments show that further pre- training LMs on these automatically generated data significantly improves LMs’ capability on spatial understanding, which in turn helps to better solve two external datasets, bAbI, and boolQ. We hope that this work can foster inves- tigations into more sophisticated models for spatial reasoning over text. 1 Introduction Spatial reasoning is a cognitive process based on the construction of mental representations for spatial objects, relations, and transforma- tions (Clements and Battista, 1992), which is necessary for many natural language understand- ing (NLU) tasks such as natural language navi- gation (Chen et al., 2019; Roman Roman et al., 2020; Kim et al., 2020), human-machine interac- tion (Landsiedel et al., 2017; Roman Roman et al., 2020), dialogue systems (Udagawa et al., 2020), and clinical analysis (Datta and Roberts, 2020). Modern language models (LM), e.g., BERT (De- vlin et al., 2019), ALBERT (Lan et al., 2020), and XLNet (Yang et al., 2019) have seen great suc- cesses in natural language processing (NLP). How- ever, there has been limited investigation into spa- tial reasoning capabilities of LMs. To the best of our knowledge, bAbI (Weston et al., 2015) (Fig 9) is the only dataset with direct textual spatial ques- tion answering (QA) (Task 17), but it is synthetic ∗Work was done while at the Allen Institute for AI. and overly simplified: (1) The underlying scenes are spatially simple, with only three objects and relations only in four directions. (2) The stories for these scenes are two short, templated sentences, each describing a single relation between two ob- jects. (3) The questions typically require up to two-steps reasoning due to the simplicity of those stories. To address these issues, this paper proposes a new dataset, SPARTQA1 (see Fig. 1). Specifically, (1) SPARTQA is built on NLVR’s (Suhr et al., 2017) images containing more objects with richer spatial structures (Fig. 1b). (2) SPARTQA’s stories are more natural, have more sentences, and richer in spatial relations in each sentence. (3) SPARTQA’s questions require deeper reasoning and have four types: find relation (FR), find blocks (FB), choose object (CO), and yes/no (YN), which allows for more fine-grained analysis of models’ capabilities. We showed annotators random images from NLVR, and instructed them to describe objects and relationships not exhaustively at the cost of …

Content truncated to 500 words