Learning to Generalize for Sequential Decision Making

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3046–3063 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 3046 Learning to Generalize for Sequential Decision Making Xusen Yin, Ralph Weischedel, and Jonathan May Information Sciences Institute University of Southern California {xusenyin,weisched,jonmay}@isi.edu Abstract We consider problems of making sequences of decisions to accomplish tasks, interacting via the medium of language. These problems are often tackled with reinforcement learning ap- proaches. We find that these models do not generalize well when applied to novel task do- mains. However, the large amount of computa- tion necessary to adequately train and explore the search space of sequential decision mak- ing, under a reinforcement learning paradigm, precludes the inclusion of large contextualized language models, which might otherwise en- able the desired generalization ability. We introduce a teacher-student imitation learning methodology and a means of converting a re- inforcement learning model into a natural lan- guage understanding model. Together, these methodologies enable the introduction of con- textualized language models into the sequen- tial decision making problem space. We show that models can learn faster and generalize more, leveraging both the imitation learning and the reformulation. Our models exceed teacher performance on various held-out deci- sion problems, by up to 7% on in-domain prob- lems and 24% on out-of-domain problems. 1 Introduction We make many decisions as we interact with the world. When we are rewarded (respectively, pun- ished), we learn to modify not only the proximal cause of the stimulus but the chain of decisions leading up to it, to encourage (respectively, discour- age) future similar results. This process naturally is the paradigm of Reinforcement Learning (RL). Policy-based learning seeks to find good estimates for Q(s, a), a function that returns the expected cu- mulative reward (known as a Q-value) if action a is chosen at state s. A desirable property of method- ologies to learn Q is their ability to generalize such that an appropriate action can be taken when en- countering a previously unseen state. Recent advances have shown strong evidence of generalization in spatiotemporal modalities such as robotic manipulation (Xu et al., 2018), video games (Tessler et al., 2017), and autonomous navigation (Zhu et al., 2017). However, in the modality of lan- guage, there is less work applying generalization approaches to decision making. Useful applica- tions of sequential decision making language mod- els are personal assistants that proactively antici- pate client needs; anti-phishing mediation agents that waste a would-be thief’s time with relevant but non-helpful responses; and investigative journal- ist assistants that determine what to read, whom to contact, and what questions to ask to create a revelatory news report. Neural reinforcement learning (RL) training ap- proaches, such as those used to play action video games (Mnih et al., 2013), have potential appli- cability in language-based decision making due to their ability to learn to navigate adversarial or exploratory scenarios. Naturally, the generaliza- tion and background knowledge capability afforded by large contextualized language models such as BERT (Devlin et al., …

Content truncated to 500 words