Estimating Semantic Similarity between In-Domain and Out-of-Domain Samples

Proceedings of the The 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023), pages 409–416 July 13-14, 2023 ©2023 Association for Computational Linguistics Estimating Semantic Similarity between In-Domain and Out-of-Domain Samples Rhitabrat Pokharel and Ameeta Agrawal Department of Computer Science Portland State University {pokharel,ameeta}@pdx.edu Abstract Prior work typically describes out-of-domain (OOD) or out-of-distribution (OODist) sam- ples as those that originate from dataset(s) or source(s) different from the training set but for the same task. When compared to in-domain (ID) samples, the models have been known to usually perform poorer on OOD samples, al- though this observation is not consistent. An- other thread of research has focused on OOD detection, albeit mostly using supervised ap- proaches. In this work, we first consolidate and present a systematic analysis of multiple defini- tions of OOD and OODist as discussed in prior literature. Then, we analyze the performance of a model under ID and OOD/OODist settings in a principled way. Finally, we seek to identify an unsupervised method for reliably identifying OOD/OODist samples without using a trained model. The results of our extensive evaluation using 12 datasets from 4 different tasks suggest the promising potential of unsupervised metrics in this task. 1 Introduction What happens when you train a machine learning model on a dataset and use it to predict a sam- ple whose source is unknown? Would you fully rely on the model’s prediction on the test sample? Basically, this situation is encountered in most real- world scenarios where the test sample may differ considerably from the training samples. Recent works show that models perform poorer on the samples that come from a different distribution (Gokhale et al., 2022). In many real-world sce- narios, such as health and law, false predictions or misclassified results could have significant conse- quences, and as such identifying out-of-domain or out-of-distribution data beforehand is critical. Previous works have defined OOD and OODist data in different ways or used them interchangeably. Early works define data that comes from a related but different domain as OOD (Dai et al., 2007), whereas OODist data has been defined as the data that might have been collected at a different time (Ovadia et al., 2019). In recent studies, (Chrysos- tomou and Aletras, 2022) use the term OOD to describe different datasets for the same task (e.g., SST, IMDb, and Yelp for sentiment classification), whereas (Lin et al., 2022) use OODist to describe the datasets that are not in the training set, includ- ing those that are subsets of the same dataset (e.g., PDTB 2.0 (Carlson et al., 2002)). In this paper, we first present a focused analysis of all the various terminologies used in this context in recent works. Another thread of research has focused on iden- tifying OOD/OODist samples, mostly through su- pervised methods (Varshney et al., 2022; Chiang and Lee, 2022; Gokhale et al., 2022). However, considering that trained models may not always be available, we take a complementary approach in this work to identify metric(s) that may be able …

Content truncated to 500 words