Don’t Copy the Teacher: Data and Model Challenges in Embodied Dialogue

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9361 - 9368 December 7-11, 2022 ©2022 Association for Computational Linguistics Don’t Copy the Teacher: Data and Model Challenges in Embodied Dialogue So Yeon Min1 Hao Zhu2 Ruslan Salakhutdinov1 Yonatan Bisk2 Machine Learning1 and Language Technologies2 at Carnegie Mellon University {soyeonm,hzhu2,rsalakhu,ybisk}@andrew.cmu.edu Abstract Embodied dialogue instruction following re- quires an agent to complete a complex se- quence of tasks from a natural language ex- change. The recent introduction of benchmarks (Padmakumar et al., 2022) raises the question of how best to train and evaluate models for this multi-turn, multi-agent, long-horizon task. This paper contributes to that conversation, by arguing that imitation learning (IL) and related low-level metrics are actually misleading and do not align with the goals of embodied dia- logue research and may hinder progress. We provide empirical comparisons of metrics, analysis of three models, and make suggestions for how the field might best progress. First, we observe that models trained with IL take spurious actions during evaluation. Second, we find that existing models fail to ground query utterances, which are essential for task comple- tion. Third, we argue evaluation should focus on higher-level semantic goals. 1 1 Introduction Dialogue is key to how humans collaborate; through dialogue, we query information, confirm our understanding, or banter in a friendly man- ner. Since communication helps us work more efficiently and successfully, it is only natural to imbue for collaborative agents with this same abil- ity. Most work has focused on grounded dialogues for embodied navigation (Thomason et al., 2020; Chi et al., 2019; Roman et al., 2020) or limited interaction (Suhr et al., 2019), which are narrower domains than the larger instruction following lit- erature (Tellex et al., 2011, 2020; Shridhar et al., 2020; Blukis et al., 2018, 2021; Min et al., 2021). The first step towards engaging in a dialogue, is being able to understand and learn from it. Pic- ture a child watching their parents with the goal 1Code to be released at https://github.com/soyeonm/TEACh_FILM to learn by imitation. They witness instructions, clarifications, mistakes, and banter. Begging the question: What should one learn from noisy natural dialogues? Unlike in alinguistic tasks where modeling hu- mans has recently proved helpful for search strate- gies (Deitke et al., 2022), we focus on language based tasks that require learning lexical-visual- action correspondences. We discuss and compare three paradigms: Instruction Following (IF), ac- tions from Entire Dialogue History (EDH) and Tra- jectory from Dialogue (TfD). The novel TEACh dataset (Padmakumar et al., 2021) proposes EDH as the primary metric and uses the Episodic Trans- former (ET) (Pashevich et al., 2021) trained with behavior cloning as their baseline. We also include comparisons to the EDH competitive Symbiote2 system and we adapt FILM (Min et al., 2021), a recent method for general IF, to dialog instruction following (DIF) on TEACH. FILM and Symbiote belong to a different family of models, focusing on abstract planning trained at a higher semantic level than behavior cloning. …

Content truncated to 500 words