Self-Attentive, Multi-Context One-Class Classification for Unsupervised Anomaly Detection on Text

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4061–4071 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 4061 Self-Attentive, Multi-Context One-Class Classification for Unsupervised Anomaly Detection on Text Lukas Ruff† Yury Zemlyanskiy‡ Robert Vandermeulen§ Thomas Schnake† Marius Kloft§‡ †Technical University of Berlin, Berlin, Germany ‡University of Southern California, Los Angeles, United States §Technical University of Kaiserslautern, Kaiserslautern, Germany {lukas.ruff,t.schnake}@tu-berlin.de yury.zemlyanskiy@usc.edu {vandermeulen,kloft}@cs.uni-kl.de Abstract There exist few text-specific methods for un- supervised anomaly detection, and for those that do exist, none utilize pre-trained models for distributed vector representations of words. In this paper we introduce a new anomaly detection method—Context Vector Data De- scription (CVDD)—which builds upon word embedding models to learn multiple sentence representations that capture multiple seman- tic contexts via the self-attention mechanism. Modeling multiple contexts enables us to per- form contextual anomaly detection of sen- tences and phrases with respect to the mul- tiple themes and concepts present in an un- labeled text corpus. These contexts in com- bination with the self-attention weights make our method highly interpretable. We demon- strate the effectiveness of CVDD quantita- tively as well as qualitatively on the well- known Reuters, 20 Newsgroups, and IMDB Movie Reviews datasets. 1 Introduction Anomaly Detection (AD) (Chandola et al., 2009; Pimentel et al., 2014; Aggarwal, 2017) is the task of discerning rare or unusual samples in a corpus of unlabeled data. A common approach to AD is one-class classification (Moya et al., 1993), where the objective is to learn a model that compactly describes “normality”—usually as- suming that most of the unlabeled training data is “normal,” i.e. non-anomalous. Deviations from this description are then deemed to be anoma- lous. Examples of one-class classification meth- ods are the well-known One-Class SVM (OC- SVM) (Sch¨olkopf et al., 2001) and Support Vector Data Description (SVDD) (Tax and Duin, 2004). Applying AD to text is useful for many appli- cations including discerning anomalous web con- tent (e.g. posts, reviews, or product descriptions), automated content management, spam detection, and characterizing news articles so as to identify similar or dissimilar novel topics. Recent work has found that proper text representation is criti- cal for designing well-performing machine learn- ing algorithms. Given the exceptional impact that universal vector embeddings of words (Bengio et al., 2003) such as word2vec (Mikolov et al., 2013), GloVe (Pennington et al., 2014), and Fast- Text (Bojanowski et al., 2017; Joulin et al., 2017) or dynamic vector embeddings of text by language models such as ELMo (Peters et al., 2018) and BERT (Devlin et al., 2018) have had on NLP, it is somewhat surprising that there has been no work on adapting AD techniques to use such unsuper- vised pre-trained models. Existing AD methods for text still typically rely on bag-of-words (BoW) text representations (Manevitz and Yousef, 2001, 2007; Mahapatra et al., 2012; Kannan et al., 2017). In this work, we introduce a novel one-class classification method that takes advantage of pre- trained word embedding models for performing AD on …

Content truncated to 500 words