Multi-Domain Goal-Oriented Dialogues (MultiDoGO): Strategies toward Curating and Annotating Large Scale Dialogue Data

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 4526–4536, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 4526 Multi-Domain Goal-Oriented Dialogues (MultiDoGO): Strategies toward Curating and Annotating Large Scale Dialogue Data Denis Peskov1∗†, Nancy Clarke†2, Jason Krone†2, Brigitta Fodor*3, Yi Zhang2, Adel Youssef2 and Mona Diab2 1University of Maryland, 2Amazon AWS AI, 3Wayfair dpeskov@cs.umd.edu, bfodor@wayfair.com {njc, kronej, yizhng, adel, diabmona}@amazon.com Abstract The need for high-quality, large-scale, goal- oriented dialogue datasets continues to grow as virtual assistants become increasingly wide- spread. However, publicly available datasets useful for this area are limited either in their size, linguistic diversity, domain coverage, or annotation granularity. In this paper, we present strategies toward curating and anno- tating large scale goal oriented dialogue data. We introduce the MultiDoGO dataset to over- come these limitations. With a total of over 81K dialogues harvested across six domains, MultiDoGO is over 8 times the size of MultiWOZ, the other largest comparable di- alogue dataset currently available to the pub- lic. Over 54K of these harvested conversa- tions are annotated for intent classes and slot labels. We adopt a Wizard-of-Oz approach wherein a crowd-sourced worker (the “cus- tomer”) is paired with a trained annotator (the “agent”). The data curation process was con- trolled via biases to ensure a diversity in dia- logue flows following variable dialogue poli- cies. We provide distinct class label tags for agents vs. customer utterances, along with applicable slot labels. We also compare and contrast our strategies on annotation granular- ity, i.e. turn vs. sentence level. Furthermore, we compare and contrast annotations curated by leveraging professional annotators vs the crowd. We believe our strategies for eliciting and annotating such a dialogue dataset scales across modalities and domains and potentially languages in the future. To demonstrate the efficacy of our devised strategies we establish neural baselines for classification on the agent and customer utterances as well as slot label- ing for each domain. ∗Work performed while at Amazon AWS AI †Denotes equal contribution 1 Introduction Modern Natural Language Understanding (NLU) frameworks for dialogues are by definition data hungry. They require large amounts of training data representative of goal oriented conversations reflecting both context and diversity. But human responses in goal-oriented dialogues are less pre- dictable than automated systems (Bordes et al., 2016). For example, “Please do this” cannot be in- terpreted without a broader context. Only by see- ing previous utterances, such as requests to book a flight on a specific day to a specific destina- tion, can this task be performed. Additionally, a single intent can be phrased in multiple ways de- pending on context; “book my flight”, “finalize my reservation”, “Yes, the 6 pm one” may all be re- ferring to a flight-booking intent. Hence, entire conversations, rather than independent utterances, must be collected. Such data is even more perti- nent to modeling NLU and related tasks as they re- quire large, varied, and ideally human-generated datasets. …

Content truncated to 500 words