Can You Put it All Together: Evaluating Conversational Agents’ Ability to Blend Skills

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2021–2030 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 2021 Can You Put it All Together: Evaluating Conversational Agents’ Ability to Blend Skills Eric Michael Smith*, Mary Williamson*, Kurt Shuster, Jason Weston, Y-Lan Boureau Facebook AI Research {ems,marywilliamson,kshuster,jase,ylan}@fb.com Abstract Being engaging, knowledgeable, and empa- thetic are all desirable general qualities in a conversational agent. Previous work has in- troduced tasks and datasets that aim to help agents to learn those qualities in isolation and gauge how well they can express them. But rather than being specialized in one single quality, a good open-domain conversational agent should be able to seamlessly blend them all into one cohesive conversational flow. In this work, we investigate several ways to com- bine models trained towards isolated capabil- ities, ranging from simple model aggregation schemes that require minimal additional train- ing, to various forms of multi-task training that encompass several skills at all training stages. We further propose a new dataset, Blended- SkillTalk, to analyze how these capabilities would mesh together in a natural conversa- tion, and compare the performance of differ- ent architectures and training schemes. Our experiments show that multi-tasking over sev- eral tasks that focus on particular capabilities results in better blended conversation perfor- mance compared to models trained on a sin- gle skill, and that both unified or two-stage ap- proaches perform well if they are constructed to avoid unwanted bias in skill selection or are fine-tuned on our new task. 1 Introduction A good open-domain conversational agent should have a well-rounded set of skills1 and qualities that allow it to seamlessly blend listening with empa- thy, providing knowledgeable responses, and talk- ing about various topics from everyday life to their favorite hobbies or latest challenges. 1”Skills” in the conversational AI literature is sometimes taken to mean a very defined specific set of abilities such as telling the weather (e.g., Zhou et al. (2020)). Our use in this paper is much more general and refers to any desirable capa- bility. Recent research has made solid strides towards gauging and improving performance of open- domain conversational agents along specific axes such as how knowledgeable they are (Dinan et al., 2019b; Moghe et al., 2018; Qin et al., 2019), how well they can display empathy (Rashkin et al., 2019; Lin et al., 2019) or talk about their personal background (Zhang et al., 2018; Li et al., 2017). However it remains unclear whether models opti- mized for performance along one of these axes can retain the learned skill while blending it with other desirable skills, or how to best conduct simultane- ous training of multiple skills. In this work, we compare several ways to com- bine tasks designed to evaluate and improve a sin- gle conversational skill, ranging from multi-task training over several datasets to training a top-level classifier to play the role of a dialogue manager and query the most appropriate single-skill pre- trained model for a response. …

Content truncated to 500 words