Deal or No Deal? End-to-End Learning of Negotiation Dialogues

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2443–2453 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Deal or No Deal? End-to-End Learning for Negotiation Dialogues Mike Lewis1, Denis Yarats1, Yann N. Dauphin1, Devi Parikh2,1 and Dhruv Batra2,1 1Facebook AI Research {mikelewis,denisy,ynd}@fb.com2Georgia Institute of Technology {parikh,dbatra}@gatech.edu Abstract Much of human dialogue occurs in semi- cooperative settings, where agents with different goals attempt to agree on com- mon decisions. Negotiations require com- plex communication and reasoning skills, but success is easy to measure, making this an interesting task for AI. We gather a large dataset of human-human negoti- ations on a multi-issue bargaining task, where agents who cannot observe each other’s reward functions must reach an agreement (or a deal) via natural language dialogue. For the first time, we show it is possible to train end-to-end models for ne- gotiation, which must learn both linguistic and reasoning skills with no annotated di- alogue states. We also introduce dialogue rollouts, in which the model plans ahead by simulating possible complete continu- ations of the conversation, and find that this technique dramatically improves per- formance. Our code and dataset are pub- licly available.1 1 Introduction Intelligent agents often need to cooperate with oth- ers who have different goals, and typically use natural language to agree on decisions. Negotia- tion is simultaneously a linguistic and a reasoning problem, in which an intent must be formulated and then verbally realised. Such dialogues contain both cooperative and adversarial elements, and re- quire agents to understand, plan, and generate ut- terances to achieve their goals (Traum et al., 2008; Asher et al., 2012). 1https://github.com/facebookresearch/ end-to-end-negotiator We collect the first large dataset of natural lan- guage negotiations between two people, and show that end-to-end neural models can be trained to negotiate by maximizing the likelihood of human actions. This approach is scalable and domain- independent, but does not model the strategic skills required for negotiating well. We fur- ther show that models can be improved by train- ing and decoding to maximize reward instead of likelihood—by training with self-play reinforce- ment learning, and using rollouts to estimate the expected reward of utterances during decoding. To study semi-cooperative dialogue, we gather a dataset of 5808 dialogues between humans on a negotiation task. Users were shown a set of items with a value for each, and asked to agree how to divide the items with another user who has a dif- ferent, unseen, value function (Figure 1). We first train recurrent neural networks to imi- tate human actions. We find that models trained to maximise the likelihood of human utterances can generate fluent language, but make comparatively poor negotiators, which are overly willing to com- promise. We therefore explore two methods for improving the model’s strategic reasoning skills— both of which attempt to optimise for the agent’s goals, rather than simply imitating humans: Firstly, instead of training to optimise likeli- hood, we show that our agents can be consider- ably improved using self …

Content truncated to 500 words