Improving Unsupervised Out-of-domain detection through Pseudo Labeling and Learning

Findings of the Association for Computational Linguistics: EACL 2023, pages 1031–1041 May 2-6, 2023 ©2023 Association for Computational Linguistics Improving Unsupervised Out-of-domain Detection through Pseudo Labeling and Learning Byounghan Lee1 Jaesik Kim1,2 Junekyu Park1,3 Kyung-Ah Sohn1,∗ Ajou University1, University of Pennsylvania2, Superb AI3 {qudgks96, kasohn}@ajou.ac.kr Jaesik.Kim@pennmedicine.upenn.edu idbluefish@gmail.com Abstract Unsupervised out-of-domain (OOD) detection is a task aimed at discriminating whether given samples are from the in-domain or not, without the categorical labels of in-domain instances. Unlike supervised OOD, as there are no la- bels for training a classifier, previous works on unsupervised OOD detection adopted the one-class classification (OCC) approach, as- suming that the training samples come from a single domain. However, in-domain instances in many real world applications can have a het- erogeneous distribution (i.e., across multiple domains or multiple classes). In this case, OCC methods have difficulty in reflecting the cate- gorical information of the domain properly. To tackle this issue, we propose a two-stage frame- work that leverages the latent categorical infor- mation to improve representation learning for textual OOD detection. In the first stage, we train a transformer-based sentence encoder for pseudo labeling by contrastive loss and cluster loss. The second stage is pseudo label learning in which the model is re-trained with pseudo- labels obtained in the first stage. The empiri- cal results on the three datasets show that our two-stage framework significantly outperforms baseline models in more challenging scenarios. 1 Introduction Deep neural networks show outstanding perfor- mance on benchmark datasets that have the same training and test domains. However, once the model is deployed to the real world, it can face out- of-domain (OOD) instances that make the model predict unreliable outcomes related to AI safety is- sues (Amodei et al., 2016; Hendrycks and Gimpel, 2017). For this reason, the OOD detection task aims to discriminate whether given instances are from in-domain (IND) or not. One of the main OOD detection approaches is to use a classifier that predicts the labels of IND samples, based on the * indicates corresponding author. fact that the classifier has lower confidence in pre- dicting the OOD samples than the IND (Hendrycks and Gimpel, 2017; Lee et al., 2018). As this approach targets only supervised tasks that require IND labels to train the classifier, it has a limitation on unsupervised tasks. To overcome this problem, recent studies have proposed unsuper- vised OOD detection (or the without label scenario) that can be utilized in a more general use case (Xu et al., 2021; Jin et al., 2022). This setting can be regarded as one-class classification (OCC) because it uses only IND instances without labels and aims to distinguish novel samples from IND instances. Within this background, unsupervised OOD de- tection methods introduce OCC approaches such as OC-SVM and SVDD (Xu et al., 2021; Sohn et al., 2020). Meanwhile, self-supervision based models exploit a novel property named inlier pri- ority (Wang et al., 2019) by using pseudo labels that are generated for surrogate supervision (e.g., rotation transformation (Hendrycks et al., …

Content truncated to 500 words