Generating Challenge Datasets for Task-Oriented Conversational Agents through Self-Play

Proceedings of Recent Advances in Natural Language Processing, pages 693–702, Varna, Bulgaria, Sep 2–4, 2019. https://doi.org/10.26615/978-954-452-056-4_081 693 Generating Challenge Datasets for Task-Oriented Conversational Agents through Self-Play Sourabh Majumdar1, Serra Sinem Tekiro˘glu2, and Marco Guerini2 1The State University of New York, NY 14260, Buffalo, USA smajumda@buffalo.edu 2Fondazione Bruno Kessler, Via Sommarive 18, Povo, Trento, Italy tekiroglu@fbk.eu, guerini@fbk.eu Abstract End-to-end neural approaches are becom- ing increasingly common in conversa- tional scenarios due to their promising performances when provided with suffi- cient amount of data. In this paper, we present a novel methodology to address the interpretability of neural approaches in such scenarios by creating challenge datasets using dialogue self-play over mul- tiple tasks/intents. Dialogue self-play al- lows generating large amount of synthetic data; by taking advantage of the com- plete control over the generation process, we show how neural approaches can be evaluated in terms of unseen dialogue pat- terns. We propose several out-of-pattern test cases each of which introduces a nat- ural and unexpected user utterance phe- nomenon. As a proof of concept, we built a single and a multiple memory net- work, and show that these two architec- tures have diverse performances depend- ing on the peculiar dialogue patterns. 1 Introduction In recent years, there has been an increasing re- search on neural approaches for conversational systems. Such approaches include the realiza- tion of full end-to-end systems (Serban et al., 2016; Bordes et al., 2017), the capacity to incor- porate or query structured knowledge sources into neural architectures (Eric and Manning, 2017), the use of zero-shot learning or of synthetic di- alogues generation techniques to mitigate effort of domain portability (Zhao and Eskenazi, 2018; Guerini et al., 2018). Although state-of-the-art neural models achieve high performances in many domains, the sheer size of data they require repre- sents a bottleneck, especially for under-resourced dialogue domains. In addition, it is hard to inter- pret their behaviour since neural models are in- trinsically opaque. In this paper we propose a novel methodology to address the aforementioned problems by synthetically creating conversation datasets with peculiar characteristics. In partic- ular, we focus on (i) scalable and portable ap- proaches for generating dialogues from scratch in- volving complex and structured knowledge, and (ii) strategies for isolating and evaluating specific reasoning capabilities of neural architectures using synthetic challenge sets. To this end, we utilize dialogue self-play strategies (Shah et al., 2018) simulating task oriented dialogues between a con- versational agent and a user. Dialogue simulation grants complete control over the generated data al- lowing building special test cases, each of which introduces a natural and unexpected user utterance phenomenon, that has never been seen at training time. It should be noted that the use of natural data would require manual annotation/selection of ex- amples for each phenomenon of interest, making this goal intractable. Another problem with current datasets is the lack of exhaustiveness: they usually focus on one specific domain intent. There are few datasets cov- ering multiple intents in the same scenario, but usually these intents …

Content truncated to 500 words