Knowing More About Questions Can Help: Improving Calibration in Question Answering

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1958–1970 August 1–6, 2021. ©2021 Association for Computational Linguistics 1958 Knowing More About Questions Can Help: Improving Calibration in Question Answering Shujian Zhang Chengyue Gong The University of Texas at AustinEunsol Choi szhang19@utexas.edu, {cygong,eunsol}@cs.utexas.edu Abstract We study calibration in question answering, es- timating whether model correctly predicts an- swer for each question. Unlike prior work which mainly rely on the model’s confidence score, our calibrator incorporates information about the input example (e.g., question and the evidence context). Together with data aug- mentation via back translation, our simple ap- proach achieves 5-10% gains in calibration ac- curacy on reading comprehension benchmarks. Furthermore, we present the first calibration study in the open retrieval setting, compar- ing the calibration accuracy of retrieval-based span prediction models and answer generation models. Here again, our approach shows con- sistent gains over calibrators relying on the model confidence. Our simple and efficient calibrator can be easily adapted to many tasks and model architectures, showing robust gains in all settings.1 1 Introduction Despite rapid progress in AI models, building a question answering (QA) system that can always correctly answer any given query is beyond our reach. Thus, questioners have to interpret the model prediction, deciding whether to trust it. We study providing an accurate estimate of the cor- rectness of model prediction for each example at test time. As making incorrect predictions can be much more costly than making no prediction (e.g., missing diagnosis is much more costly than query- ing human experts), calibrators can bring practical benefits (Kamath et al., 2020). Existing work on calibration focuses on mdoel confidence, such as the max probability of the pre- dicted class (Guo et al., 2017; Desai and Durrett, 2020). Unlike classification tasks, question an- swering explores large output space, either through 1Code is available at https://github.com/ szhang42/Calibration_qa. answer generation (Raffel et al., 2020; Lewis et al., 2020) or selecting a span from provided docu- ments (Rajpurkar et al., 2016). In both settings, op- timal decoding is often prohibitively expensive, and heuristic decoding is a standard practice (Seo et al., 2017). Thus, relying on the model’s confidence score alone is not sufficient for calibration (Kumar and Sarawagi, 2019). Nonetheless, prior work (Kamath et al., 2020; Jagannatha and Yu, 2020) relied heavily on model confidence, such as the max probability of the pre- dicted answer, together with a handful of manually crafted features containing little information about the input, such as the length of the question. We empower the calibrator by introducing an input example embedding from a pre-trained language model (Alberti et al., 2019; Liu et al., 2019) fine- tuned on QA supervision data as additional features. With this simple and general feature, calibrator can identify questions regarding rare entities or exam- ples with little lexical overlap between the question and the context. We bring further gains by para- phrasing questions or contexts respectively through back translation (Sennrich et al., 2016), providing lexical variations of the question and the context …

Content truncated to 500 words