Systematic Generalization on gSCAN: What is Nearly Solved and What is Next?

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2180–2188 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 2180 Systematic Generalization on gSCAN: What is Nearly Solved and What is Next? Linlu Qiu 1∗ Hexiang Hu 1 † Bowen Zhang 2 Peter Shaw 1 Fei Sha 1 1 Google Research 2 USC Abstract We analyze the grounded SCAN (gSCAN) benchmark, which was recently proposed to study systematic generalization for grounded language understanding. First, we study which aspects of the original benchmark can be solved by commonly used methods in multi- modal research. We find that a general- purpose Transformer-based model with cross- modal attention achieves strong performance on a majority of the gSCAN splits, sur- prisingly outperforming more specialized ap- proaches from prior work. Furthermore, our analysis suggests that many of the remaining errors reveal the same fundamental challenge in systematic generalization of linguistic con- structs regardless of visual context. Second, inspired by this finding, we propose challeng- ing new tasks for gSCAN by generating data to incorporate relations between objects in the vi- sual environment. Finally, we find that current models are surprisingly data inefficient given the narrow scope of commands in gSCAN, suggesting another challenge for future work. 1 Introduction Systematic generalization refers to the ability to un- derstand new compositions of previously observed concepts and linguistic constructs. While humans exhibit this ability, neural networks often strug- gle. To study systematic generalization, several synthetic datasets have been proposed. Lake and Baroni (2018) introduced SCAN, a dataset com- posed of natural language instructions paired with action sequences, and splits in various ways to as- sess systematic generalization. Recently, Ruis et al. (2020) introduced the grounded SCAN (gSCAN) benchmark, which similarly pairs natural language instructions with action sequences, but further re- quires that instructions are interpreted within the context of a grid-based visual navigation environ- ment. In this work, we analyze which aspects of ∗Work done as a Google AI Resident. †Work done while at USC. gSCAN can currently be solved by general-purpose models, and propose new tasks and evaluation met- rics for future research of systematic generalization on gSCAN. First, to understand which aspects of gSCAN can be addressed by a general-purpose approach, we evaluate a Transformer-based model with cross- modal attention. Cross-modal attention has been proven effective for other multi-modal tasks (Lu et al., 2019; Tan and Bansal, 2019; Chen et al., 2020). It achieves strong performance on a majority of the splits, surprisingly outperform- ing several “specialist” approaches designed for gSCAN (Heinze-Deml and Bouchacourt, 2020; Gao et al., 2020; Kuo et al., 2020). We analyze the remaining errors, finding that many of the er- rors appear to be related to the same fundamental challenge in systematic generalization of linguis- tic constructs studied in datasets such as SCAN, regardless of the visual context. Our analysis motivates the creation of an addi- tional gSCAN task, which features a greater degree of complexity in how natural language instructions are grounded in the visual context. In this …

Content truncated to 500 words