ALFRED-L: Investigating the Role of Language for Action Learning in Interactive Visual Environments

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9369 - 9378 December 7-11, 2022 ©2022 Association for Computational Linguistics ALFRED-L: Investigating the Role of Language for Action Learning in Interactive Visual Environments Arjun R. Akula1∗, Spandana Gella2, Aishwarya Padmakumar2, Mahdi Namazifar2, Mohit Bansal2,3, Jesse Thomason2,4, Dilek Hakkani-Tür2 1Google AI, 3University of North Carolina at Chapel Hill 2Amazon Alexa AI, 4University of Southern California arjunakula@google.com, sgella@amazon.com, padmakua@amazon.com mahdinam@amazon.com, mbansal@cs.unc.edu, jessetho@usc.edu, hakkanit@amazon.com Abstract Embodied Vision and Language Task Comple- tion requires an embodied agent to interpret nat- ural language instructions and egocentric visual observations to navigate through and interact with environments. In this work, we examine ALFRED (Shridhar et al., 2020), a challenging benchmark for embodied task completion, with the goal of gaining insight into how effectively models utilize language. We find evidence that sequence-to-sequence and transformer-based models trained on this benchmark are not suffi- ciently sensitive to changes in input language instructions. Next, we construct a new test split – ALFRED-L to test whether ALFRED mod- els can generalize to task structures not seen during training that intuitively require the same types of language understanding required in ALFRED. Evaluation of existing models on ALFRED-L suggests that (a) models are overly reliant on the sequence in which objects are visited in typical ALFRED trajectories and fail to adapt to modifications of this sequence and (b) models trained with additional augmented trajectories are able to adapt relatively better to such changes in input language instructions. 1 Introduction Recently a number of benchmark datasets have been proposed to study the ability of embodied agents to understand natural language in the con- text of egocentric visual observations and predict sequences of executable actions to answer ques- tions (Das et al., 2018), navigate to desired destina- tions (Anderson et al., 2018; Chen et al., 2019), or additionally manipulate objects to complete tasks (Shridhar et al., 2020; Padmakumar et al., 2021). Although multi-modal transformer-based mod- els have achieved tremendous progress on many of these datasets (Zhu et al., 2021b; Hong et al., 2021; ∗ Work done in part while AA was research intern at Amazon Alexa AI in Summer 2021. Low-level Instructions: <I1>: Walk to the coffee maker on the end of the counter. <I2>: Grab the spoon from the counter. <I3>: Go to the stove and focus on the top left burner. <I4>: Place the spoon in the pan. <I5>: Turn around and go to the kitchen sink <I1>: Walk to the coffee maker on the end of the counter. <I2>: Grab the spoon from the counter. <I3>: Go to the stove and focus on the top left burner. <I4>: Place the spoon in the pan. <I5>: Turn around and go to the kitchen sink. <I5-r>: Go back to the stove and focus on the top left burner. <I1>: Walk to the coffee maker on the end of the counter. <I3>: Go to the stove and focus on the top left burner. <I5>: Turn around and go to the kitchen sink. …

Content truncated to 500 words