Language Models are Few-Shot Butlers

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9312–9318 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 9312 Language Models are Few-Shot Butlers Vincent Micheli University of Geneva vincent.micheli@unige.ch François Fleuret University of Geneva francois.fleuret@unige.ch Abstract Pretrained language models demonstrate strong performance in most NLP tasks when fine-tuned on small task-specific datasets. Hence, these autoregressive models constitute ideal agents to operate in text-based envi- ronments where language understanding and generative capabilities are essential. Nonethe- less, collecting expert demonstrations in such environments is a time-consuming endeavour. We introduce a two-stage procedure to learn from a small set of demonstrations and further improve by interacting with an environment. We show that language models fine-tuned with only 1.2% of the expert demonstrations and a simple reinforcement learning algorithm achieve a 51% absolute improvement in success rate over existing methods in the ALFWorld environment. 1 Introduction Over the past few years, successive generations of language models (Radford et al., 2018, 2019; Brown et al., 2020) have reshaped the way we approach Natural Language Processing problems. These Transformer-based (Vaswani et al., 2017) networks scale to ever-increasing amounts of pa- rameters, data and compute (Kaplan et al., 2020) while demonstrating impressive transfer-learning capabilities across a wide variety of benchmarks (Wang et al., 2018, 2019; Rajpurkar et al., 2018; Reddy et al., 2019). However, the application of modern language models to action generation in text-based environ- ments (Côté et al., 2018; Hausknecht et al., 2019; Shridhar et al., 2021) remains largely unexplored (Yao et al., 2020). Intuitively, large-scale models leveraging strong linguistic priors should thrive in settings where observations and actions are textual (Luketina et al., 2019), but several issues arise. In- deed, agents must conform to an environment’s gen- erative grammar and gameplay specificities making Goal: Rinse the egg to put it in the microwave. Obs: Looking quickly around you, you see a cabinet, a garbagecan, a coffeemachine, [...], a stoveburner, a sinkbasin and a microwave. Action: go to sinkbasin Obs: You arrive at sinkbasin. You see a but- terknife, a potato, a spoon and a tomato. Action: go to garbagecan Obs: You arrive at garbagecan. You see an egg. Action: take egg from garbagecan Obs: You pick up the egg from the garbagecan. Action: go to sinkbasin Obs: You arrive at sinkbasin. You see a but- terknife, a potato, a spoon and a tomato. Action: clean egg with sinkbasin Obs: You clean the egg using the sinkbasin. Action: go to microwave Obs: You arrive at microwave. The microwave is closed. Action: open microwave Obs: You open the microwave. Action: put egg in/on microwave Figure 1: Example of a human-annotated, out-of- distribution task instance solved by GPT⋆ partial. off the shelf transfer fail in most games. A way to overcome this problem is to acquire expert demon- strations and resort to the widely used paradigm of fine-tuning on task-specific data (Howard and Ruder, 2018; Radford et al., 2018; Devlin et al., 2019). Nevertheless, collecting demonstrations in text-based environments requires far more …

Content truncated to 500 words