Wizard of Tasks: A Novel Conversational Dataset for Solving Real-World Tasks in Conversational Settings

Proceedings of the 29th International Conference on Computational Linguistics, pages 3514–3529 October 12–17, 2022. 3514 Wizard of Tasks: A Novel Conversational Dataset for Solving Real-World Tasks in Conversational Settings Jason Ingyu Choi, Saar Kuzi, Nikhita Vedula, Jie Zhao, Giuseppe Castellucci, Marcus Collins, Shervin Malmasi, Oleg Rokhlenko and Eugene Agichtein Amazon {chojson,skuzi,veduln,zhaozjie,giusecas}@amazon.com {collmr,malmasi,olegro,eugeneag}@amazon.com Abstract Conversational Task Assistants (CTAs) are conversational agents whose goal is to help hu- mans perform real-world tasks. CTAs can help in exploring available tasks, answering task- specific questions and guiding users through step-by-step instructions. In this work, we present Wizard of Tasks, the first cor- pus of such conversations in two domains: Cooking and Home Improvement. We crowd- sourced a total of 549 conversations (18,077 utterances) with an asynchronous Wizard-of- Oz setup, relying on recipes from WholeFoods Market for the cooking domain, and WikiHow articles for the home improvement domain. We present a detailed data analysis and show that the collected data can be a valuable and challenging resource for CTAs in two tasks: In- tent Classification (IC) and Abstractive Ques- tion Answering (AQA). While on IC we ac- quired a high performing model (≥85% F1), on AQA the performance is far from being satisfactory (∼27% BertScore-F1), suggesting that more work is needed to solve the task of low-resource AQA. 1 Introduction In recent years, the way to access web information has changed from using keyword- and semantics- based search (Manning et al., 2008), to Question Answering (QA) (Chen et al., 2017) and Conver- sational Agents (CAs) (Radlinski and Craswell, 2017). CAs have evolved to support different types of interaction and information: there are CAs for chatting (Zhou et al., 2020) and CAs that let users interact with existing information systems to ac- complish specific tasks, e.g., booking a restaurant, as in task-oriented dialogues (Bobrow et al., 1977; Wen et al., 2017). A specific type of information humans are look- ing for is how to perform tasks, e.g., cooking a dish or fixing a household problem. The Web con- tains articles accompanied by images or videos with step-by-step instructions to perform variegate tasks. Existing CAs can be used mainly to browse tasks or to answer specific questions. However, they fail in providing a comprehensive natural con- versation that includes search, context-aware QA, step-by-step instructions, and multi-modal informa- tion sharing. The Alexa Prize TaskBot1 Challenge (Gottardi et al., 2022) is a research challenge sponsored by Amazon to foster research on CTAs to assist hu- mans in executing real-world tasks. The targeted tasks require multiple steps and decisions, includ- ing multi-modal (voice and visual) user experi- ences. The challenge includes two domains: Cook- ing, i.e., guiding people in preparing recipes; and Home Improvement, i.e., guiding people through common household do-it-yourself tasks such as painting a wall or pruning trees. As shown in Fig- ure 1, CTAs should support QA capabilities on Web sources as well as selected recipes or articles, dia- log management to support step-by-step instruction navigation, and multi-modal interaction. In this paper, we present Wizard of Tasks2 …

Content truncated to 500 words