FlowPrior: Learning Expressive Priors for Latent Variable Sentence Models

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 267–277 June 6–11, 2021. ©2021 Association for Computational Linguistics 267 DATE: Detecting Anomalies in Text via Self-Supervision of Transformers Andrei Manolache1,2 and Florin Brad1 and Elena Burceanu1,2,3 1Bitdefender 2University of Bucharest, Romania 3Institute of Mathematics of the Romanian Academy {amanolache, fbrad, eburceanu}@bitdefender.com Abstract Leveraging deep learning models for Anomaly Detection (AD) has seen widespread use in re- cent years due to superior performances over traditional methods. Recent deep methods for anomalies in images learn better features of normality in an end-to-end self-supervised set- ting. These methods train a model to discrimi- nate between different transformations applied to visual data and then use the output to com- pute an anomaly score. We use this approach for AD in text, by introducing a novel pretext task on text sequences. We learn our DATE model end-to-end, enforcing two independent and complementary self-supervision signals, one at the token-level and one at the sequence- level. Under this new task formulation, we show strong quantitative and qualitative results on the 20Newsgroups and AG News datasets. In the semi-supervised setting, we outperform state-of-the-art results by +13.5% and +6.9%, respectively (AUROC). In the unsupervised configuration, DATE surpasses all other meth- ods even when 10% of its training data is con- taminated with outliers (compared with 0% for the others). 1 Introduction Anomaly Detection (AD) can be intuitively defined as the task of identifying examples that deviate from the other ones to a degree that arouses sus- picion (Hawkins, 1980). Research into AD spans several decades (Chandola et al., 2009; Aggarwal, 2015) and has proved fruitful in several real-world problems, such as intrusion detection systems (Ban- oth et al., 2017), credit card fraud detection (Dor- ronsoro et al., 1997), and manufacturing (Kam- merer et al., 2019). Our DATE method is applicable in the semi- supervised AD setting, in which we only train on clean, labeled normal examples, as well as the unsu- pervised AD setting, where both unlabeled normal and abnormal data are used for training. Typical deep learning approaches in AD involve learning features of normality using autoencoders (Hawkins et al., 2002; Sakurada and Yairi, 2014; Chen et al., 2017) or generative adversarial networks (Schlegl et al., 2017). Under this setup, anomalous exam- ples lead to a higher reconstruction error or differ significantly compared with generated samples. Recent deep AD methods for images learn more effective features of visual normality through self- supervision, by training a deep neural network to discriminate between different transformations ap- plied to the input images (Golan and El-Yaniv, 2018; Wang et al., 2019). An anomaly score is then computed by aggregating model predictions over several transformed input samples. We adapt those self-supervised classifica- tion methods for AD from vision to learn anomaly scores indicative of text normality. ELECTRA (Clark et al., 2020) proposes an effi- cient language representation learner, which solves the Replaced Token Detection (RTD) task. Here the input tokens are plausibly …

Content truncated to 500 words