Realistic Conversational Question Answering with Answer Selection based on Calibrated Confidence and Uncertainty Measurement

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 477–490 May 2-6, 2023 ©2023 Association for Computational Linguistics Realistic Conversational Question Answering with Answer Selection based on Calibrated Confidence and Uncertainty Measurement Soyeong Jeong1 Jinheon Baek2 Sung Ju Hwang1,2 Jong C. Park1∗ School of Computing1 Graduate School of AI2 Korea Advanced Institute of Science and Technology1,2 {starsuzi,jinheon.baek,sjhwang82,jongpark}@kaist.ac.kr Abstract Conversational Question Answering (ConvQA) models aim at answering a question with its rel- evant paragraph and previous question-answer pairs that occurred during conversation multi- ple times. To apply such models to a real-world scenario, some existing work uses predicted an- swers, instead of unavailable ground-truth an- swers, as the conversation history for inference. However, since these models usually predict wrong answers, using all the predictions with- out filtering significantly hampers the model performance. To address this problem, we propose to filter out inaccurate answers in the conversation history based on their estimated confidences and uncertainties from the Con- vQA model, without making any architectural changes. Moreover, to make the confidence and uncertainty values more reliable, we pro- pose to further calibrate them, thereby smooth- ing the model predictions. We validate our models, Answer Selection-based realistic Con- versation Question Answering, on two stan- dard ConvQA datasets, and the results show that our models significantly outperform rele- vant baselines. Code is available at: https: //github.com/starsuzi/AS-ConvQA. 1 Introduction Conversational Question Answering (ConvQA) is the task of answering a series of questions during conversation, taking into account a given relevant paragraph (Choi et al., 2018; Reddy et al., 2019). Contrary to traditional extractive question answer- ing tasks (Rajpurkar et al., 2016; Trischler et al., 2017) that answer each question with the given paragraph just once, ConvQA aims at answering the current question using its previous question- answer pairs taking into account the given para- graph multiple times. For example, as illustrated in Figure 1, the goal of ConvQA is to correctly answer the question Q3 based on the previous conversation ∗ Corresponding author : H.E. Pennypacker : Van Nostrand 1: What were some of his pseudonyms? C : Like the other three characters, Kramer has pseudonyms he uses in various schemes; (𝐴1) H.E. Pennypacker, Dr. Martin van Nostrand, and Professor Peter van Nostrand are the most popular. Under the name H.E. Pennypacker in (𝐴2) “The Puerto Rican Day” (𝐴3) Kramer poses as a prospective buyer interested in an elegant apartment in order to use its bathroom. … He also uses the Van Nostrand alias in the episode "The Slicer", posing as a "Juilliard-trained dermatologist" … 2 : Which episode did he use this name in? : What happened in this episode? : Kramer : Van Nostrand : posing as a "Juilliard-trained dermatologist" : Kramer poses as a prospective buyer 55.76 55.44 57.35 54 55 56 57 58 All Pred. No Pred. AS-ConvQA : posing as a "Juilliard-trained dermatologist" Figure 1: Illustration of realistic ConvQA evaluation with three models: 1) using all predicted answers (All Pred.); 2) not using predicted …

Content truncated to 500 words