Supervised Seeded Iterated Learning for Interactive Language Learning

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 3962–3970, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 3962 Supervised Seeded Iterated Learning for Interactive Language Learning Yuchen Lu Mila University of Montreal Soumye Singhal Mila University of Montreal Florian Strub DeepMind Olivier Pietquin Google Research Brain Team Aaron Courville Mila University of Montreal, CIFAR Abstract Language drift has been one of the major ob- stacles to train language models through in- teraction. When word-based conversational agents are trained towards completing a task, they tend to invent their language rather than leveraging natural language. In recent liter- ature, two general methods partially counter this phenomenon: Supervised Selfplay (S2P) and Seeded Iterated Learning (SIL). While S2P jointly trains interactive and supervised losses to counter the drift, SIL changes the training dynamics to prevent language drift from occurring. In this paper, we first high- light their respective weaknesses, i.e., late- stage training collapses and higher negative likelihood when evaluated on human corpus. Given these observations, we introduce Super- vised Seeded Iterated Learning (SSIL) to com- bine both methods to minimize their respective weaknesses. We then show the effectiveness of SSIL in the language-drift translation game. 1 Introduction Since the early days of NLP (Winograd, 1971), conversational agents have been designed to in- teract with humans through language to solve di- verse tasks, e.g., remote instructions (Thomason et al., 2015) or booking assistants (Bordes et al., 2017; El Asri et al., 2017). In this goal-oriented dialogue setting, the conversational agents are of- ten designed to compose with predefined language utterances (Lemon and Pietquin, 2007; Williams et al., 2014; Young et al., 2013). Even if such approaches are efficient, they also tend to narrow down the agent’s language diversity. To remove this restriction, recent work has been exploring in- teractive word-based training. In this setting, the agents are generally trained through a two-stage process (Wei et al., 2018; De Vries et al., 2017; Shah et al., 2018; Li et al., 2016a; Das et al., 2017): Firstly, the agent is pretrained on a human-labeled corpus through supervised learning to generate grammatically reasonable sentences. Secondly, the agent is finetuned to maximize the task-completion score by interacting with a user. Due to sample- complexity and reproducibility issues, the user is generally replaced by a game simulator that may evolve with the conversational agent. Unfortu- nately, this pairing may lead to the language drift phenomenon, where the conversational agents grad- ually co-adapt, and drift away from the pretrained natural language. The model thus becomes unfit to interact with humans (Chattopadhyay et al., 2017; Zhu et al., 2017; Lazaridou et al., 2020). While domain-specific methods exist to counter language drift (Lee et al., 2019; Li et al., 2016b), a simple task-agnostic method consists of combining interactive and supervised training losses on a pre- training corpus (Wei et al., 2018; Lazaridou et al., 2016), which was later formalized as Supervised SelfPlay (S2P) (Lowe et al., 2020). Inspired by language evolution and cultural trans- mission (Kirby, 2001; …

Content truncated to 500 words