What makes a good conversation? How controllable attributes affect human judgments

Proceedings of NAACL-HLT 2019, pages 1702–1723 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 1702 What makes a good conversation? How controllable attributes affect human judgments Abigail See∗ Stanford University abisee@stanford.edu Stephen Roller Facebook AI Research roller@fb.com Douwe Kiela Facebook AI Research dkiela@fb.com Jason Weston Facebook AI Research jase@fb.com Abstract A good conversation requires balance – be- tween simplicity and detail; staying on topic and changing it; asking questions and an- swering them. Although dialogue agents are commonly evaluated via human judgments of overall quality, the relationship between quality and these individual factors is less well-studied. In this work, we examine two controllable neural text generation methods, conditional training and weighted decoding, in order to control four important attributes for chitchat dialogue: repetition, specificity, response-relatedness and question-asking. We conduct a large-scale human evaluation to measure the effect of these control parame- ters on multi-turn interactive conversations on the PersonaChat task. We provide a detailed analysis of their relationship to high-level as- pects of conversation, and show that by con- trolling combinations of these variables our models obtain clear improvements in human quality judgments. 1 Introduction Neural generation models for dialogue, despite their ubiquity in current research, are still poorly understood. Well known problems, such as the genericness and repetitiveness of responses (Ser- ban et al., 2016a), remain without a de facto solu- tion. Strikingly, the factors that determine human judgments of overall conversation quality are al- most entirely unexplored. Most works have been limited to the next utterance prediction problem, whereas a multi-turn evaluation is necessary to evaluate the quality of a full conversation. In this work we both (i) conduct a large-scale study to identify the fine-grained factors governing human judgments of full conversations, and (ii) develop models that apply our findings in practice, ∗A.S. completed most of this work at Facebook (FAIR). Figure 1: We manipulate four low-level attributes and measure their effect on human judgments of individual conversational aspects, as well as overall quality. leading to state-of-the-art performance. Specifi- cally, we identify and study eight aspects of con- versation that can be measured by human judg- ments, while varying four types of low-level at- tributes that can be algorithmically controlled in neural models; see Figure 1. To control the low- level model attributes, we consider two simple but general algorithms: conditional training, in which the neural model is conditioned on additional con- trol features, and weighted decoding, in which control features are added to the decoding scoring function at test time only. One major result of our findings is that existing work has ignored the importance of conversational flow, as standard models (i) repeat or contradict previous statements, (ii) fail to balance specificity with genericness, and (iii) fail to balance asking questions with other dialogue acts. Conducting experiments on the PersonaChat task (Zhang et al., 2018b), we obtain significantly higher engaging- ness scores than the baseline by optimizing con- trol of repetition, specificity and question-asking over multiple turns. Using these findings, our …

Content truncated to 500 words