Action-Based Conversations Dataset: A Corpus for Building More In-Depth Task-Oriented Dialogue Systems

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3002–3017 June 6–11, 2021. ©2021 Association for Computational Linguistics 3002 Action-Based Conversations Dataset: A Corpus for Building More In-Depth Task-Oriented Dialogue Systems Derek Chen†, Howard Chen†, Yi Yang†, Alexander Lin†, Zhou Yu‡ †ASAPP, New York, NY 10007 ‡Columbia University, NY †{dchen,hchen,yyang,alin}@asapp.com, zy2461@columbia.edu Abstract Existing goal-oriented dialogue datasets focus mainly on identifying slots and values. How- ever, customer support interactions in reality often involve agents following multi-step pro- cedures derived from explicitly-defined com- pany policies as well. To study customer ser- vice dialogue systems in more realistic set- tings, we introduce the Action-Based Con- versations Dataset (ABCD), a fully-labeled dataset with over 10K human-to-human di- alogues containing 55 distinct user intents requiring unique sequences of actions con- strained by policies to achieve task success. We propose two additional dialog tasks, Ac- tion State Tracking and Cascading Dialogue Success, and establish a series of baselines in- volving large-scale, pre-trained language mod- els on this dataset. Empirical results demon- strate that while more sophisticated networks outperform simpler models, a considerable gap (50.8% absolute accuracy) still exists to reach human-level performance on ABCD. 1 1 Introduction The broad adoption of virtual assistants and cus- tomer service chatbots in recent years has been driven in no small part by the usefulness of these tools, whereby actions are taken on behalf of the user to accomplish their desired targets (Ama- zon, 2019; Google, 2019). Research into task- oriented dialogue has concurrently made tremen- dous progress on natural language understanding of user needs (Wu et al., 2019; Rastogi et al., 2020b; Liang et al., 2020). However, selecting actions in real life requires not only obeying user requests, but also following practical policy limi- tations which may be at odds with those requests. For example, while a user may ask for a refund on their purchase, an agent should only honor such a request if it is valid with regards to the store’s re- turn policy. Described in actions, before an agent 1All code and data will be available at this location. Figure 1: An interaction from ABCD (left) starts with the customer receiving a prompt (top right) to ground the dialogue. The agent follows the guidelines (bottom right) to identify the customer intent and to assist them in resolving the issue through a series of actions. can [Offer Refund], they must first [Validate Pur- chase]. Furthermore, resolving customer issues often concerns multiple actions completed in suc- cession with a specific order since prior steps may influence future decision states. (See Figure 1) To more closely model real customer service agents, we present the Action-Based Conversa- tions Dataset (ABCD) consisting of 10,042 con- versations containing numerous actions with pre- cise procedural requirements. These actions dif- fer from typical dialogue acts because tracking them necessitates striking a balance between ex- ternal user requests and internally-imposed guide- lines. Thus, the major difference between ABCD and other dialogue datasets, such as …

Content truncated to 500 words