Taskmaster-1: Toward a Realistic and Diverse Dialog Dataset

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 4516–4525, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 4516 Taskmaster-1: Toward a Realistic and Diverse Dialog Dataset Bill Byrne1∗ Karthik Krishnamoorthi1∗ Chinnadhurai Sankar2∗ Arvind Neelakantan1 Daniel Duckworth1 Semih Yavuz3 Ben Goodrich1 Amit Dubey1 Andy Cedilnik1 Kyu-Young Kim1 1Google LLC, Mountain View, California 2Mila, Universit´e de Montr´eal 3University of California, Santa Barbara {billb,krishnamoorthi,aneelakantan}@google.com Abstract A significant barrier to progress in data-driven approaches to building dialog systems is the lack of high quality, goal-oriented conversa- tional data. To help satisfy this elementary requirement, we introduce the initial release of the Taskmaster-1 dataset which includes 13,215 task-based dialogs comprising six do- mains. Two procedures were used to create this collection, each with unique advantages. The first involves a two-person, spoken “Wiz- ard of Oz” (WOz) approach in which trained agents and crowdsourced workers interact to complete the task while the second is “self- dialog” in which crowdsourced workers write the entire dialog themselves. We do not re- strict the workers to detailed scripts or to a small knowledge base and hence we observe that our dataset contains more realistic and di- verse conversations in comparison to existing datasets. We offer several baseline models in- cluding state of the art neural seq2seq architec- tures with benchmark performance as well as qualitative human evaluations. Dialogs are la- beled with API calls and arguments, a simple and cost effective approach which avoids the requirement of complex annotation schema. The layer of abstraction between the dialog model and the service provider API allows for a given model to interact with multiple ser- vices that provide similar functionally. Finally, the dataset will evoke interest in written vs. spoken language, discourse patterns, error han- dling and other linguistic phenomena related to dialog system research, development and de- sign.1 1 Introduction Voice-based “personal assistants” such as Apple’s SIRI, Microsoft’s Cortana, Amazon Alexa, and the ∗Equal Contribution 1Dataset available at https://g.co/dataset/taskmaster-1 Google Assistant have finally entered the main- stream. This development is generally attributed to major breakthroughs in speech recognition and text-to-speech (TTS) technologies aided by recent progress in deep learning (Lecun et al., 2015), ex- ponential gains in compute power (Steinkrau et al., 2005; Jouppi et al., 2017), and the ubiquity of powerful mobile devices. The accuracy of ma- chine learned speech recognizers (Hinton et al., 2012) and speech synthesizers (van den Oord et al., 2016) are good enough to be deployed in real-world products and this progress has been driven by pub- licly available labeled datasets. However, conspic- uously absent from this list is equal progress in machine learned conversational natural language understanding (NLU) and generation (NLG). The NLU and NLG components of dialog systems start- ing from the early research work (Weizenbaum, 1966) to the present commercially available per- sonal assistants largely rely on rule-based systems. The NLU and NLG systems are often carefully programmed for very narrow and specific cases …

Content truncated to 500 words