PLACES: Prompting Language Models for Social Conversation Synthesis

Findings of the Association for Computational Linguistics: EACL 2023, pages 844–868 May 2-6, 2023 ©2023 Association for Computational Linguistics PLACES: Prompting Language Models for Social Conversation Synthesis Maximillian Chen1∗, Alexandros Papangelis2, Chenyang Tao2, Seokhwan Kim2, Andy Rosenbaum2, Yang Liu2, Zhou Yu1, Dilek Hakkani-Tur2 1Columbia University, 2Amazon Alexa AI maxchen@cs.columbia.edu, zy2461@columbia.edu {papangea,chenyt,seokhwk,andros,yangliud,hakkanit}@amazon.com Abstract Collecting high quality conversational data can be very expensive for most applications and infeasible for others due to privacy, ethical, or similar concerns. A promising direction to tackle this problem is to generate synthetic dialogues by prompting large language mod- els. In this work, we use a small set of expert- written conversations as in-context examples to synthesize a social conversation dataset us- ing prompting. We perform several thorough evaluations of our synthetic conversations com- pared to human-collected conversations. This includes various dimensions of conversation quality with human evaluation directly on the synthesized conversations, and interactive hu- man evaluation of chatbots fine-tuned on the synthetically generated dataset. We addition- ally demonstrate that this prompting approach is generalizable to multi-party conversations, providing potential to create new synthetic data for multi-party tasks. Our synthetic multi-party conversations were rated more favorably across all measured dimensions compared to conversa- tion excerpts sampled from a human-collected multi-party dataset. 1 Introduction Training dialogue models typically requires an abundance of data, as with any machine learning task. However, collecting high quality data is dif- ficult and expensive, especially for dialogue tasks where there often is no “right answer” when devel- oping the trajectory of a conversation. Typically di- alogue data are sourced from crowdworkers and the quality of annotations, evaluations, and conversa- tions can vary considerably (Zhao and Zhu, 2014), often necessitating guardrails such as credential- based worker selection or defensive task design for quality control (Allahbakhsh et al., 2013). To accommodate data scarcity in training dia- logue tasks, low resource methods have become ∗Work done during internship at Amazon Alexa AI Yeah, I grew up there. Have you ever been to Ithaca? Yeah, I lived there for a long time. Go in the summer: swimming in waterfalls! It's gorge-ous. Sorry, had to do it. Ha. I have always meant to visit Ithaca, I hear it is a pretty cool town. Speaker 1 Speaker 2 Speaker 3 Speaker 1 Speaker 2 Speaker 3 Speaker 2 That's cool. I've always wanted to visit the UK. I'm from a small village in the UK. It's really old and has a lot of history. Speaker 1 Speaker 2 It's a great place to visit. Speaker 1 What's the best thing about your hometown? Speaker 2 Is your city a historical place? Speaker 1 Not really. 200 years ago, it was just a small insignificant village. Speaker 2 How did it grow into such a large place? Speaker 1 Large deposits of coal were found nearly and so many industries located themselves here. The village quickly grew into a key industrial centre. Speaker 2 What's it like? Speaker 2 It's a really great place to live. The people are …

Content truncated to 500 words