The Negochat Corpus of Human-agent Negotiation Dialogues

The Negochat Corpus of Human-agent Negotiation Dialogues Vasily Konovalov1, Ron Artstein2, Oren Melamud1, Ido Dagan1 1Computer Science Department, Bar-Ilan University, Israel 2USC Institute for Creative Technologies, USA vaskonov@yahoo.com, artstein@ict.usc.edu, melamuo@cs.biu.ac.il, dagan@cs.biu.ac.il Abstract Annotated in-domain corpora are crucial to the successful development of dialogue systems of automated agents, and in particular for developing natural language understanding (NLU) components of such systems. Unfortunately, such important resources are scarce. In this work, we introduce an annotated natural language human-agent dialogue corpus in the negotiation domain. The corpus was collected using Amazon Mechanical Turk following the ‘Wizard-Of-Oz’ approach, where a ‘wizard’ human translates the participants’ natural language utterances in real time into a semantic language. Once dialogue collection was completed, utterances were annotated with intent labels by two independent annotators, achieving high inter-annotator agreement. Our initial experiments with an SVM classifier show that automatically inferring such labels from the utterances is far from trivial. We make our corpus publicly available to serve as an aid in the development of dialogue systems for negotiation agents, and suggest that analogous corpora can be created following our methodology and using our available source code. To the best of our knowledge this is the first publicly available negotiation dialogue corpus. Keywords: dialogue systems, negotiation corpora, crowdsourcing 1. Introduction Annotated in-domain corpora are crucial to the successful development of dialogue systems of automated agents, and in particular for developing natural language understanding (NLU) components of such systems (Lasecki et al., 2013). While NLU is considered to be a challenging task, previ- ous studies in the Human Computer Interactions field have found that an important goal of any human-agent dialogue system should be an intuitive interface with interaction en- vironments as real and natural as possible. This highlights the importance of achieving good natural language capa- bilities for automated agents, which go beyond traditional menu-based interactions (Coen, 1998). Negotiation is an important task in our daily lives. It con- cerns anything from a mundane ‘time to go to lunch’, to our salary, and to issues that can have dramatic effects on the lives of millions (Musambachime and Hopmann, 2001). As such, various automated agents have been created to negotiate with people in different settings with varied fac- tors, such as the number of parties, number of interactions and number of issues to be negotiated. (Zuckerman et al., 2013). In particular, the popularity of the negotiation domain led to the establishment of the international Automated Negoti- ating Agents Competition (ANAC) Workshop. In this pop- ular competition human-agent dialogues are currently re- stricted to a fixed set of menu-based interactions (Baarslag et al., 2012), while extension to natural language interac- tions is on the future roadmap. Since 2010, the competition has focused on agents that are based on the open Genius environment (Lin et al., 2012). Although natural language annotated corpora are consid- ered an important resource for training and tuning NLU components, to the best of our knowledge there are no such publicly available negotiation corpora. In this work we address …

Content truncated to 500 words