“None of the Above”: Measure Uncertainty in Dialog Response Retrieval

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2013–2020 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 2013 “None of the Above”: Measure Uncertainty in Dialog Response Retrieval Yulan Feng, Shikib Mehri, Maxine Eskenazi and *Tiancheng Zhao Language Technologies Institute, Carnegie Mellon University *SOCO.AI {yulanf,amehri,max}@cs.cmu.edu *tianchez@soco.ai Abstract This paper discusses the importance of uncov- ering uncertainty in end-to-end dialog tasks and presents our experimental results on uncer- tainty classification on the processed Ubuntu Dialog Corpus1. We show that instead of re- training models for this specific purpose, we can capture the original retrieval model’s un- derlying confidence concerning the best pre- diction using trivial additional computation. 1 Introduction Uncertainty modeling is a widely explored problem in dialog research. Stochastic models like deep Q- networks (Tegho et al., 2017), Gaussian processes (Gai and Young, 2014), and partially observable Markov decision process (Roy et al., 2000) are often used in spoken dialog systems to optimize dialog management by explicitly estimating uncer- tainty in policy assignments. However, these approaches are either computa- tionally intensive (Gal and Ghahramani, 2015) or require significant work on refining policy repre- sentations (Gai and Young, 2014). Moreover, most current uncertainty studies in dialog focus on the dialog management component. End-to-end (E2E) dialog retrieval models jointly encode a dialog and a candidate response (Wu et al., 2016; Zhou et al., 2018), assuming the ground truth is always present in the candidate set, which is not the case in pro- duction. Larson et al. (2019) recently showed that classifiers that perform well on in-scope intent clas- sification for task-oriented dialog systems strug- gle to identify out-of-scope queries. The response selection task in the most recent Dialog System Technology Challenge (Chulaka Gunasekara and Lasecki, 2019) also explicitly mentions that “none 1Our datasets for the NOTA task are released at https://github.com/yfeng21/nota prediction of the proposed utterances is a good candidate” should be a valid option. The goal of this paper is to set a new direction for future task-oriented dialog system research: while retrieving the best candidate is crucial, it should be equally important to identify when the correct response (i.e. ground truth) is not present in the candidate set. In this paper, we measure the E2E re- trieval model’s capability to capture uncertainty by inserting an additional “none of the above” (NOTA) candidate into the proposed response set at infer- ence time. The contributions of this paper include: (1) demonstrating that it is crucial to learn the rela- tionship amongst the candidates as a set instead of looking at point-wise matching to solve the NOTA detection task. As a result, the logistic regres- sion (LogReg) approach proposed here consistently achieves the best performance compared to several strong baselines. (2) extensive experiments show that the raw output score (logits) is more informa- tive in terms of representing model confidence than normalized probabilities after the Softmax layer. 2 Related Work Our use of NOTA to measure uncertainty in dia- log response is motivated by the design …

Content truncated to 500 words