Dialogue in the Wild: Learning from a Deployed Role-Playing Game with Humans and Bots

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 611–624 August 1–6, 2021. ©2021 Association for Computational Linguistics 611 Dialogue in the Wild: Learning from a Deployed Role-Playing Game with Humans and Bots Kurt Shuster∗ Jack Urbanek∗ Emily Dinan Arthur Szlam Jason Weston Facebook AI Research Abstract Much of NLP research has focused on crowd- sourced static datasets and the supervised learning paradigm of training once and then evaluating test performance. As argued in de Vries et al. (2020), crowdsourced data has the issues of lack of naturalness and relevance to real-world use cases, while the static dataset paradigm does not allow for a model to learn from its experiences of using language (Silver et al., 2013). In contrast, one might hope for machine learning systems that become more useful as they interact with people. In this work, we build and deploy a role-playing game, whereby human players converse with learning agents situated in an open-domain fantasy world. We show that by training mod- els on the conversations they have with hu- mans in the game the models progressively im- prove, as measured by automatic metrics and online engagement scores. This learning is shown to be more efficient than crowdsourced data when applied to conversations with real users, as well as being far cheaper to collect. 1 Introduction Humans learn to use language over the course of their lives from the interactions they have with the world and other people. Yet, the prevailing dominant paradigm in natural language process- ing (NLP) research is to build a fixed dataset from which to train a model and then freeze it, without any ability for the model to interact with humans us- ing language at training time at all. While we need such interaction in order to study human-machine communication to its full extent, constraints usu- ally inhibit such research. Firstly, conducting such experiments can be costly. Many datasets in NLP are collected with crowdsourcing, whereby one pays the crowdworkers to perform interaction and ∗* Equal Contribution. annotation tasks. This leads to several issues, not least that research budgets for paying crowdwork- ers mean that data will have a limit. Secondly, as crowdworkers are motivated by pay, not by interest in the tasks themselves, the data distribution may not match the desired one (de Vries et al., 2020). In this work we study the ability of dialogue agents in an open world1 to iteratively learn from conversations with intrinsically motivated humans. In order to engage humans at scale, we build and de- ploy a (free to play) game with a purpose (Von Ahn, 2006) whereby human players role-play characters and converse with other characters (that are our learning models) situated within the game world. We choose a rich fantasy game world, in order to maximize engagement. Our system iterates be- tween collecting data of human-model interactions, retraining updated models on the newly collected data, and redeploying them. Simultaneously, it provides a natural metric to evaluate and compare models online using human continue …

Content truncated to 500 words