CoLo: A Contrastive Learning Based Re-ranking Framework for One-Stage Summarization

Proceedings of the 29th International Conference on Computational Linguistics, pages 608–615 October 12–17, 2022. 608 Distribution Calibration for Out-of-Domain Detection with Bayesian Approximation Yanan Wu1∗, Zhiyuan Zeng1∗, Keqing He2∗, Yutao Mou1 , Pei Wang1, Weiran Xu1∗ 1Beijing University of Posts and Telecommunications, Beijing, China 2Meituan Group, Beijing, China {yanan.wu,zengzhiyuan,myt,wangpei,xuweiran}@bupt.edu.cn {hekeqing}@meituan.com Abstract Out-of-Domain (OOD) detection is a key com- ponent in a task-oriented dialog system, which aims to identify whether a query falls outside the predefined supported intent set. Previous softmax-based detection algorithms are proved to be overconfident for OOD samples. In this paper, we analyze overconfident OOD comes from distribution uncertainty due to the mis- match between the training and test distribu- tions, which makes the model can’t confidently make predictions thus probably causing abnor- mal softmax scores. We propose a Bayesian OOD detection framework to calibrate distri- bution uncertainty using Monte-Carlo Dropout. Our method is flexible and easily pluggable into existing softmax-based baselines and gains 33.33% OOD F1 improvements with increas- ing only 0.41% inference time compared to MSP. Further analyses show the effectiveness of Bayesian learning for OOD detection. 1 1 Introduction Detecting Out-of-Domain (OOD) or unknown in- tents from user queries is key for a task-oriented dialog system (Gnewuch et al., 2017; Akasaki and Kaji, 2017; Tulshan and Dhage, 2018; Shum et al., 2018; Zeng et al., 2021a,b; Wu et al., 2022). It aims to know when a user query falls outside their range of predefined supported intents to avoid per- forming wrong operations. Different from normal intent classification tasks, lack of labeled OOD ex- amples leads to poor prior knowledge about these unknown intents, making it challenging to detect OOD samples. A rich line of OOD intent detection algo- rithms has been developed recently, among which softmax-based methods demonstrated promise (Guo et al., 2017; Liang et al., 2018; Zheng et al., ∗The first three authors contribute equally. Weiran Xu is the corresponding author. 1Our code is available at https://github.com/ pris-nlp/COLING2022_Bayesian-for-OOD/. 0.0 0.5 0.0 0.5 Probability 0.0 0.5 0 30 60 90 120 150 (a) Different output distributions. 0.0 0.5 0 30 60 90 120 150 (b) Averaged output distribution. (labels) 0.0 0.5 Probability threshold Figure 1: Multiple predicted probability distributions of an OOD sample under different random seeds. We train four identical models on the same data but only use different random seeds. Fig (a) displays each output distribution of an OOD input and Fig (b) shows the averaged output distribution. 2020). Softmax-based methods leverage softmax outputs extracted from an in-domain (IND) intent model and operate under the assumption that the test OOD samples get a lower likelihood proba- bility than the ID data. For example, Maximum Softmax Probability (MSP) (Hendrycks and Gim- pel, 2017) detects a test query as OOD if its max softmax probability is lower than a fixed thresh- old. However, all these models make a strong distributional assumption of the practical OOD probability being uniform, which has been proven wrong because neural networks can produce over- confidently high softmax scores even for …

Content truncated to 500 words