Increasing Faithfulness in Knowledge-Grounded Dialogue with Controllable Features

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 704–718 August 1–6, 2021. ©2021 Association for Computational Linguistics 704 Increasing Faithfulness in Knowledge-Grounded Dialogue with Controllable Features Hannah Rashkin David Reitter Gaurav Singh Tomar Dipanjan Das Google Research, New York, NY {hrashkin, reitter, gtomar, dipanjand}@google.com Abstract Knowledge-grounded dialogue systems are in- tended to convey information that is based on evidence provided in a given source text. We discuss the challenges of training a generative neural dialogue model for such systems that is controlled to stay faithful to the evidence. Existing datasets contain a mix of conversa- tional responses that are faithful to selected ev- idence as well as more subjective or chit-chat style responses. We propose different evalu- ation measures to disentangle these different styles of responses by quantifying the informa- tiveness and objectivity. At training time, ad- ditional inputs based on these evaluation mea- sures are given to the dialogue model. At generation time, these additional inputs act as stylistic controls that encourage the model to generate responses that are faithful to the pro- vided evidence. We also investigate the us- age of additional controls at decoding time us- ing resampling techniques. In addition to au- tomatic metrics, we perform a human evalu- ation study where raters judge the output of these controlled generation models to be gener- ally more objective and faithful to the evidence compared to baseline dialogue systems. 1 Introduction Dialogue systems that strive to be informative teachers are difficult to build, despite recent progress in training end-to-end systems that mimic human language at a linguistic level. These systems benefit from vast training data and great representa- tional capacity; yet there are no controls (or training objectives) available that ensure they are truthful. A more limited goal for a system is to be faithful to one or more source documents that we implicitly trust. Such a system might help educate users about a particular topic through conversational interac- tion, or it might augment a task-oriented dialogue system by providing additional information about I visit animal shelters fairly often A "no-kill" shelter is an animal shelter that does not kill healthy or treatable animals even when the shelter is full, reserving euthanasia for terminally ill animals ... Personal Experience Grounding Document There's a really nice no-kill shelter by my house that I volunteer at on wednesdays! I see. Is a no kill shelter different from regular animal shelter? Information Supported by the Grounding Document Yes, a "no-kill" shelter is an animal shelter that does not kill healthy or treatable animals even when the shelter is full. User Different response styles Figure 1: Excerpt from Wizard of Wikipedia (Dinan et al., 2019) conversation. This grounded dialogue in- cludes responses containing subjective or personal ex- periences as well as responses sharing information sup- ported by external documents. the process involved in, say, adding a new home automation device. We assume that multi-turn con- versational interaction …

Content truncated to 500 words