More Diverse Dialogue Datasets via Diversity-Informed Data Collection

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4958–4968 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 4958 More Diverse Dialogue Datasets via Diversity-Informed Data Collection Katherine Stasaski1, Grace Hui Yang2, and Marti A. Hearst1 1UC Berkeley 2Georgetown University 1{katie stasaski, hearst}@berkeley.edu 2huiyang@cs.georgetown.edu Abstract Automated generation of conversational dia- logue using modern neural architectures has made notable advances. However, these mod- els are known to have a drawback of often pro- ducing uninteresting, predictable responses; this is known as the diversity problem. We in- troduce a new strategy to address this problem, called Diversity-Informed Data Collection. Unlike prior approaches, which modify model architectures to solve the problem, this method uses dynamically computed corpus- level statistics to determine which conversa- tional participants to collect data from. Diversity-Informed Data Collection produces significantly more diverse data than baseline data collection methods, and better results on two downstream tasks: emotion classification and dialogue generation. This method is gen- eralizable and can be used with other corpus- level metrics. 1 Introduction It is well-documented that neural dialogue mod- els struggle with generating engaging, relevant re- sponses (Li et al., 2016a) and often produce ba- nal responses such as “Yeah.” While this may be an appropriate response to a chitchat conversation, to keep a human participant engaged, diversity of responses is important. Diverse models vary the language used and the content referenced, and the generated utterances differ from the most typical conversation responses some proportion of the time. A model which only generates “Yeah,” “No,” and “I don’t know” is not diverse and is not be engaging to converse with. Past work has improved model diversity with innovation on model architectures and decoding strategies (Li et al., 2016a; Baheti et al., 2018; Li et al., 2017; Shao et al., 2017; Cao and Clark, 2017; Serban et al., 2017; Zhao et al., 2017). We build upon this work to propose a novel method to col- lect and determine more diverse data to train these models with. Our method can be used in conjunc- tion with existing generation-specific model inno- vations. Some prior work on data collection processes has prioritized diversity. For instance, Rashkin et al. (2019) prompts crowdworkers to choose an underused emotion class to generate dialogue. This work encourages coverage of emotion classes, but does not consider the likelihood that some crowd- workers are better at producing certain types of data than others. This paper introduces Diversity-Informed Data Collection (DIDC), a new strategy for creating a dataset of conversational utterances via selecting which participants’ data to include in the collec- tion. The strategy progressively builds up a more diverse sub-corpus from an existing larger collec- tion. The main idea is to grow the sub-corpus by adding conversations sequentially and to assess the contribution of a new participant’s utterances to the diversity of the entire sub-corpus. This strat- egy is also applicable to on-the-fly collection of new datasets via crowdworking or similar methods. We implement DIDC with three diversity metrics: …

Content truncated to 500 words