How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering

How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering Zhengbao Jiang†, Jun Araki‡, Haibo Ding‡, Graham Neubig† †Languages Technologies Institute, Carnegie Mellon University, United States ‡Bosch Research, United States {zhengbaj,gneubig}@cs.cmu.edu {jun.araki,haibo.ding}@us.bosch.com Abstract Recent works have shown that language mod- els (LM) capture different types of knowledge regarding facts or common sense. However, because no model is perfect, they still fail to provide appropriate answers in many cases. In this paper, we ask the question, ‘‘How can we know when language models know, with con- fidence, the answer to a particular query?’’ We examine this question from the point of view of calibration, the property of a probabilistic model’s predicted probabilities actually being well correlated with the probabilities of cor- rectness. We examine three strong generative models—T5, BART, and GPT-2—and study whether their probabilities on QA tasks are well calibrated, finding the answer is a rela- tively emphatic no. We then examine methods to calibrate such models to make their confidence scores correlate better with the likelihood of correctness through fine-tuning, post-hoc pro- bability modification, or adjustment of the pre- dicted outputs or inputs. Experiments on a diverse range of datasets demonstrate the ef- fectiveness of our methods. We also perform analysis to study the strengths and limitations of these methods, shedding light on further improvements that may be made in methods for calibrating LMs. We have released the code at https://github.com/jzbjyb /lm-calibration. 1 Introduction Language models (LMs; Church, 1988; Bengio et al., 2003; Radford et al., 2019) learn to model the probability distribution of text, and in doing so capture information about various aspects of the syntax or semantics of the language at hand. Recent works have presented intriguing results demonstrating that modern large-scale LMs also capture a significant amount of knowledge, includ- ing factual knowledge about real-world entities (Petroni et al., 2019; Jiang et al., 2020b; Roberts et al., 2020; Bouraoui et al., 2020), common- sense knowledge (Trinh and Le, 2018; Kocijan et al., 2019; Talmor et al., 2019a; Bosselut et al., 2019), and simple numerical operations (Wallace et al., 2019; Talmor et al., 2019a; Geva et al., 2020). Notably, large models trained on massive crawls of Internet text (such as T5 [Raffel et al., 2019] and GPT-3 [Brown et al., 2020]) have been shown to be able to perform quite sophisticated knowledge-based tasks simply through prompting the model to predict the next words given a par- ticular cue. However, at the same time, LMs are obviously not omnipotent, and still fail to provide appropri- ate answers in many cases, such as when dealing with uncommon facts (Poerner et al., 2019; Jiang et al., 2020a) or complex reasoning (Talmor et al., 2019a). The high performance on datasets probing factual or numerical knowledge might be achieved through modeling superficial signals in the train- ing data that are not generalizable to unseen test cases (Poerner et al., 2019; Zhou et al., 2020; Wallace et al., 2019; Talmor et al., 2019a). Thus, if such …

Content truncated to 500 words