Crisscrossed Captions: Extended Intramodal and Intermodal Semantic Similarity Judgments for MS-COCO

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 300–325 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 300 Recipes for Building an Open-Domain Chatbot Stephen Roller Emily Dinan Naman Goyal Da Ju Mary Williamson Yinhan Liu∗ Jing Xu Myle Ott Kurt Shuster Eric M. Smith Y-Lan Boureau Jason Weston Facebook AI Research New York, NY {roller, edinan, namangoyal, daju, marywilliamson, jingxu23, myleott, kshuster, ems, ylan, jase}@fb.com Abstract Building open-domain chatbots is a challeng- ing area for machine learning research. While prior work has shown that scaling neural mod- els in the number of parameters and the size of the data they are trained on gives improved results, we highlight other ingredients. Good conversation requires blended skills: provid- ing engaging talking points, and displaying knowledge, empathy and personality appropri- ately, while maintaining a consistent persona. We show that large scale models can learn these skills when given appropriate training data and choice of generation strategy. We build variants of these recipes with 90M, 2.7B and 9.4B parameter models, and make our models and code publicly available. Human evaluations show our best models outperform existing approaches in multi-turn dialogue on engagingness and humanness measurements. We then discuss the limitations of this work by analyzing failure cases of our models. 1 Introduction This work provides recipes for building open- domain chatbots that perform well, i.e., to study and identify which methods work well together in order to maximize human evaluations. It has been shown across the field of NLP (Devlin et al., 2019) and in conversational agents in particular (Dinan et al., 2020; Zhang et al., 2019; Adiwar- dana et al., 2020) that pre-training on large corpora is important. Beyond simply scaling models, the two main takeaways from our study are that: (1) Large improvements can be made by fine-tuning on data that emphasizes desirable conversational skills. We select tasks that make the model focus on personality and engagingness, knowledge, and empathy, achieving large gains by using the re- cently introduced Blended Skill Talk (BST) set-up ∗Work done while at Facebook; currently AI2 Incubator. (Smith et al., 2020), which targets those aspects by providing training data and initial conversational context (personas and topics). Small models using BST can match or outperform larger models that do not. (2) Two models with the same perplexity but different decoding algorithms can give vastly different results, e.g. we show that the length of the bot’s utterances are crucial to human judgments of quality. We show, contrary to previous work which reports that beam search is inferior to sampling (Holtzman et al., 2019; Adiwardana et al., 2020), that careful choice of search hyperparameters can give strong results, e.g., constraining the minimum beam length gives a crucial control of the bland versus spicy spectrum of responses. In human evaluations our best model outper- forms both DialogGPT (Zhang et al., 2019) and the state of the art Meena chatbot (Adiwardana et al., 2020), the latter in a pairwise comparison 75% …

Content truncated to 500 words