Linguistically-Informed Specificity and Semantic Plausibility for Dialogue Generation

Proceedings of NAACL-HLT 2019, pages 3456–3466 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 3456 Linguistically-Informed Specificity and Semantic Plausibility for Dialogue Generation Wei-Jen Ko1 Greg Durrett1 Junyi Jessy Li2 1 Department of Computer Science 2 Department of Linguistics The University of Texas at Austin wjko@utexas.edu, gdurrett@cs.utexas.edu, jessy@austin.utexas.edu Abstract Sequence-to-sequence models for open- domain dialogue generation tend to favor generic, uninformative responses. Past work has focused on word frequency-based approaches to improving specificity, such as penalizing responses with only common words. In this work, we examine whether specificity is solely a frequency-related notion and find that more linguistically-driven speci- ficity measures are better suited to improving response informativeness. However, we find that forcing a sequence-to-sequence model to be more specific can expose a host of other problems in the responses, including flawed discourse and implausible semantics. We rerank our model’s outputs using externally- trained classifiers targeting each of these identified factors. Experiments show that our final model using linguistically motivated specificity and plausibility reranking improves the informativeness, reasonableness, and grammatically of responses. 1 Introduction Since the pioneering work in machine trans- lation (Sutskever et al., 2014), sequence-to- sequence (SEQ2SEQ) models have led much recent progress in open-domain dialogue generation, es- pecially single-turn generation where the input is a prompt and the output is a response. However, SEQ2SEQ methods are known to favor universal responses, e.g., “I don’t know what you are talking about” (Sordoni et al., 2015; Serban et al., 2016; Li et al., 2016a). These responses tend to be “safe” responses to many input queries, yet they usually fail to provide useful information. One promising line of research tackling this issue is to improve the specificity of responses, building on the intuition that generic responses frequently appear in the training data or consist of frequent words (Yao et al., 2016; Zhang et al., 2018b; Liu et al., 2018). However, past work in sentence specificity—the “quality of belong- ing or relating uniquely to a particular subject”1— has shown that word frequency is only one as- pect of specificity, and that specificity involves a wide range of phenomena including word us- age, sentence structure (Louis and Nenkova, 2011; Li and Nenkova, 2015; Lugini and Litman, 2017) and discourse context (Dixon, 1987; Lassonde and O’Brien, 2009). Frequency-based specificity also does not exactly capture “the amount of in- formation” as an information-theoretic concept. Hence, in dialogue generation, we can potentially make progress by incorporating more linguisti- cally driven measures of specificity, as opposed to relying solely on frequency. We present a sequence-to-sequence dialogue model that factors out specificity and explicitly conditions on it when generating a response. The decoder takes as input categorized values of sev- eral specificity metrics, embeds them, and uses them at each stage of decoding. During training, the model can learn to associate different speci- ficity levels with different types of responses. At test time, we set the specificity level to its maxi- mum value to force specific responses, which we found to be …

Content truncated to 500 words