Reducing Conversational Agents’ Overconfidence Through Linguistic Calibration

Reducing Conversational Agents’ Overconfidence Through Linguistic Calibration Sabrina J. Mielke1,2 Arthur Szlam2 Emily Dinan2 Y-Lan Boureau2 1Department of Computer Science, Johns Hopkins University, USA 2Facebook AI Research, USA sjmielke@jhu.edu {aszlam,edinan,ylan}@fb.com Abstract While improving neural dialogue agents’ fac- tual accuracy is the object of much research, another important aspect of communication, less studied in the setting of neural dialogue, is transparency about ignorance. In this work, we analyze to what extent state-of-the-art chit-chat models are linguistically calibrated in the sense that their verbalized expression of doubt (or confidence) matches the likelihood that the model’s responses are factually incor- rect (or correct). We find that these models are poorly calibrated, yet we show that likelihood of correctness can accurately be predicted. By incorporating such metacognitive features into the training of a controllable generation model, we obtain a dialogue agent with greatly improved linguistic calibration. 1 Introduction Neural generative open-domain English-language dialogue agents have made progress towards the ability to carry on chit-chat conversations with hu- mans (Adiwardana et al., 2020; Roller et al., 2021). Recent models—trained on large swaths of data from the Internet to mimic human–human con- versations—can name their favorite sports teams, describe what it’s like to be the owner of two dogs, or share their opinions on tacos. However, ask a state-of-the-art chatbot ‘‘Which is heavier, 1 kg feathers or 1 kg stone?’’, and it might confidently answer: ‘‘Feathers, because they are heavier than a kilogram of any other material.’’1 This amusing overconfidence can become problematic if some- one genuinely doesn’t know the answer and is misled into believing something false. Genera- tive chit-chat dialogue agents have many issues going beyond inaccurate answers (Xu et al., 2020; Bender et al., 2021), making them currently generally unsuitable for applications other than entertainement and research. Nevertheless, better 1Answer generated by BST 2.7B (Roller et al., 2021). control of the alignment between the confidence of an answer and its likelihood of being correct seems like a promising type of remediation: It makes models more transparent about their limi- tations directly in the dialogue rather than through extrinsic instructions for adequate use that people might overlook or forget. This goal applies Grice’s maxim of quality (Grice, 1975) on a metacogni- tive level, namely, being truthful about what one knows. Here, this would mean that if we can train accurate predictors of correctness from in- formation available to the model (input words and internal representations), then model gener- ations should convey that information. The skill of handling uncertainty would be desirable even if accuracy on factual questions ever became per- fect: Some questions do not have known answers, or have answers that depend on a context that a dialogue agent cannot know, making it perilous to ‘‘ignore ignorance" (Smithson, 2012; Ravetz, 1993). In this work, we seek to understand whether a model’s verbalized expression of confidence (‘‘Obviously, . . .’’) or doubt (‘‘I’m not sure, but. . .’’) in its answer—which we refer to throughout as linguistic confidence—corresponds to the likelihood that the answer …

Content truncated to 500 words