Modular Networks for Compositional Instruction Following

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1033–1040 June 6–11, 2021. ©2021 Association for Computational Linguistics 1033 Modular Networks for Compositional Instruction Following Rodolfo Corona Daniel Fried Coline Devin Dan Klein Trevor Darrell UC Berkeley {rcorona,dfried,coline,klein,trevordarrell}@berkeley.edu Abstract Standard architectures used in instruction fol- lowing often struggle on novel compositions of subgoals (e.g. navigating to landmarks or picking up objects) observed during training. We propose a modular architecture for follow- ing natural language instructions that describe sequences of diverse subgoals. In our ap- proach, subgoal modules each carry out nat- ural language instructions for a specific sub- goal type. A sequence of modules to execute is chosen by learning to segment the instruc- tions and predicting a subgoal type for each segment. When compared to standard, non- modular sequence-to-sequence approaches on ALFRED (Shridhar et al., 2020), a challeng- ing instruction following benchmark, we find that modularization improves generalization to novel subgoal compositions, as well as to envi- ronments unseen in training. 1 Introduction Work on grounded instruction following (MacMa- hon et al., 2006; Vogel and Jurafsky, 2010; Tellex et al., 2011; Chen and Mooney, 2011; Artzi and Zettlemoyer, 2013) has recently been driven by sequence-to-sequence models (Mei et al., 2016; Hermann et al., 2017), which allow end-to-end grounding of linguistically-rich instructions into equally-rich visual contexts (Misra et al., 2018; Anderson et al., 2018; Chen et al., 2019). These sequence-to-sequence models are monolithic: they consist of a single network structure which is ap- plied identically to every example in the dataset. Monolithic instruction following models typi- cally perform well when evaluated on test data from the same distribution seen during training. However, they often struggle in compositional gen- eralization: composing atomic parts, such as ac- tions or goals, where the parts are seen in training but their compositions are not (Lake and Baroni, 2018; Ruis et al., 2020; Hill et al., 2020). Turn left to move to the counter... Pick up the pan to the right of the toaster. Turn around to move in front of the sink. Place the pan in the sink ... Carry the pan ... Place the pan on the back of the counter by the wall. Training Test Turn around and walk past the bed... Grab the cellphone off of the cabinet. Turn right and walk ... Place the cellphone on the dresser. Turn around ... Grab the cellphone off of the chair there. Turn left and walk ... Place the cellphone down ... Figure 1: At evaluation time, an instruction follow- ing agent may need to generalize both to novel chains of subgoals encountered during training as well as to completely new environments. In the generalization condition above, the agent must generalize to multi- ple pickup actions (in green) at test time, whereas only single ones were seen at training, as well as to a new house. We propose a modular architecture to handle these cases. In this work, we improve compositional gen- …

Content truncated to 500 words