Latent Compositional Representations Improve Systematic Generalization in Grounded Question Answering

Latent Compositional Representations Improve Systematic Generalization in Grounded Question Answering Ben Bogin1 Sanjay Subramanian2 Matt Gardner2 Jonathan Berant1,2 1Tel-Aviv University 2Allen Institute for AI {ben.bogin,joberant}@cs.tau.ac.il, {sanjays,mattg}@allenai.org Abstract Answering questions that involve multi-step reasoning requires decomposing them and using the answers of intermediate steps to reach the final answer. However, state-of-the- art models in grounded question answering often do not explicitly perform decomposition, leading to difficulties in generalization to out- of-distribution examples. In this work, we propose a model that computes a representa- tion and denotation for all question spans in a bottom-up, compositional manner using a CKY-style parser. Our model induces latent trees, driven by end-to-end (the answer) supervision only. We show that this induc- tive bias towards tree structures dramatically improves systematic generalization to out-of- distribution examples, compared to strong baselines on an arithmetic expressions bench- mark as well as on CLOSURE, a dataset that focuses onsystematicgeneralization for grounded ques- tion answering. On this challenging dataset, our model reaches an accuracy of 96.1%, significantly higher than prior models that almost perfectly solve the task on a random, in-distribution split. 1 Introduction Humans can effortlessly interpret new language utterances, if they are composed of previously observed primitives and structure (Fodor and Pylyshyn, 1988). Neural networks, conversely, do not exhibit this systematicity: Although they generalize well to examples sampled from the dis- tribution of the training set, they have been shown to struggle in generalizing to out-of-distribution (OOD) examples that contain new compositions in grounded question answering (Bahdanau et al., 2019a,b) and semantic parsing (Finegan-Dollak et al., 2018; Keysers et al., 2020). Consider the question in Figure 1. This question requires query- ingobjectsizes, comparingcolors, identifyingspa- tial relations, and computing intersections between sets of objects. Neural models succeed when these concepts are combined in ways that were seen at training time. However, they commonly fail when concepts are combined in new ways at test time. A possible reason for this phenomenon is the expressivity of architectures such as LSTMs (Hochreiter and Schmidhuber, 1997) and Trans- formers (Vaswani et al., 2017), where ‘‘global’’ representations that depend on the entire input are computed. Contextualizing token representa- tions using the entire utterance potentially lets the model avoid step-by-step reasoning, ‘‘collapse’’ reasoning steps, and rely on shortcuts (Jiang and Bansal, 2019; Subramanian et al., 2019, 2020). Such failures are revealed when evaluating models for systematic generalization on OOD examples. This stands in contrast to pre-neural log-linear models, where hierarchical representations were explicitly constructed over the input (Zettlemoyer and Collins, 2005; Liang et al., 2013). In this work, we propose a model for visual question answering (QA) that, analogous to these classical pre-neural models, computes for every span in the input question a representation and a denotation, that is, the set of objects in the image that the span refers to (see Figure 1). Denotations for long spans are recursively computed from shorter spans using a bottom–up CKY-style parser without access to the entire input, leading to an inductive bias that encourages compositional com- putation. Because training …

Content truncated to 500 words