Multitasking Inhibits Semantic Drift

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5351–5366 June 6–11, 2021. ©2021 Association for Computational Linguistics 5351 Multitasking Inhibits Semantic Drift Athul Paul Jacob apjacob@mit.edu MIT CSAIL Mike Lewis mikelewis@fb.com Facebook AI Research Jacob Andreas jda@mit.edu MIT CSAIL Abstract When intelligent agents communicate to ac- complish shared goals, how do these goals shape the agents’ language? We study the dynamics of learning in latent language poli- cies (LLPs), in which instructor agents gener- ate natural-language subgoal descriptions and executor agents map these descriptions to low- level actions. LLPs can solve challenging long-horizon reinforcement learning problems and provide a rich model for studying task- oriented language use. But previous work has found that LLP training is prone to seman- tic drift (use of messages in ways inconsis- tent with their original natural language mean- ings). Here, we demonstrate theoretically and empirically that multitask training is an effec- tive counter to this problem: we prove that multitask training eliminates semantic drift in a well-studied family of signaling games, and show that multitask training of neural LLPs in a complex strategy game reduces drift and while improving sample efficiency. 1 Introduction A major goal in the study of artificial and natu- ral intelligence is to understand how language can scaffold more general problem-solving skills (e.g. Spelke, 2017), and how these skills in turn shape language itself (e.g. Gibson et al., 2017). In NLP and machine learning, latent language policies (LLPs; Andreas et al., 2018) provide a standard framework for studying these questions. An LLP consists of instructor and executor subpolicies: the instructor generates natural language messages (e.g. high-level commands or subgoals), and the execu- tor maps these messages to sequences of low-level actions (Fig. 1). LLPs have been used to construct interactive agents capable of complex reasoning (e.g. programming by demonstration) and planning over long horizons (e.g. in strategy games; Hu et al., 2019). They promise an effective and interpretable interface between planning and control. Instructor for Task 2 (build spearmen to win) build a dragon attack with spearman Multitask Executor attack with dragon send a peasant to explore Instructor for Task 1 (build dragons to win) build a spearman Single-task Executor Instructor for Task 1 (build dragons to win) executor attacks with spearman executor explores with peasant executor builds a dragon Standard training Multitask training (a) (b) (c) (d) Figure 1: In latent language policies, instructor agents (a) send natural-language commands (b) to executor agents (c), which execute them in an interactive envi- ronment (d). Jointly trained instructor–executor pairs learn to use messages in ways inconsistent with their natural language meanings (top shows a real message– action pair from a model described in Section 4). We show that multitask training with a population of task- specific instructors stabilizes message semantics and in some cases improves model performance. However, they present a number of challenges for training. As LLPs employ a human-specified space of high-level commands, they must be initial- …

Content truncated to 500 words