Can Language Models Help in System Security? Investigating Log Anomaly Detection using BERT

Can Language Models Help in System Security? Investigating Log Anomaly Detection using BERT Crispin Almodovar1 Fariza Sabrina1 Sarvnaz Karimi2 Salahuddin Azad1 1Central Queensland University, Australia 2CSIRO Data61, Sydney, Australia crispin.almodovar@cqumail.com {f.sabrina,s.azad}@cqu.edu.au {sarvnaz.karimi}@csiro.au Abstract The log files generated by networked computer systems contain valuable information that can be used to monitor system security and stabil- ity. Transformer-based natural language pro- cessing methods have proven effective in de- tecting anomalous activities from system logs. The current approaches, however, have limited practical application because they rely on log templates which cannot handle variability in log content, or they require supervised train- ing to be effective. We propose a novel log anomaly detection approach named LogFiT. It utilises a pretrained BERT-based language model and fine-tunes it towards learning the linguistic structure of system logs. The LogFiT model is trained in a self-supervised manner us- ing normal log data only. Using masked token prediction and centroid distance minimisation as training objectives, the LogFiT model learns to recognise the linguistic patterns associated with the normal log data. During inference, a discriminator function uses the LogFiT model’s top-k token prediction accuracy and computed centroid distance to determine if the input is normal or anomaly. Our experiments on three different datasets show that LogFiT is effective. 1 Introduction Cybercrime costs businesses billions of dollars an- nually (RiskIQ, 2019; Australia Department of Home Affairs, 2020; International Business Ma- chines, 2022). Log anomaly detection helps to pro- tect businesses’ digital infrastructure from cyber- attacks by providing the ability to detect abnormal activities, such as network intrusions, from large volumes of event logs generated by networked com- puter systems. Recently, approaches based on Deep Learning and Natural Language Processing (NLP) have been applied to address the log anomaly detection prob- lem. A review of the literature indicates that Long Short-Term Memory (LSTM), represented by the DeepLog model (Du et al., 2017), and Transform- ers, represented by the LogBERT model (Guo et al., 2021), are the deep learning architectures used in the state of the art research in this do- main. A practical consideration in log anomaly detection using deep learning is the availability of labeled data to be used in training predictive models. Because of the high cost of preparing la- beled data, classification-based approaches such as LogSy (Nedelkoski et al., 2020) are of limited value in production settings. Thus, a majority of log anomaly detection approaches focus on the zero-positive training scenario, in which predic- tive models are trained in a self-supervised man- ner using normal log data only (Le and Zhang). Further, Yuan et al. (2021) identifies two general categories of self-supervised models for anomaly detection: (1) forecasting-based, which attempts to predict the next log entry given previous log entries; and (2) reconstruction-based, which re- composes log sequences that have been intention- ally corrupted. The DeepLog model adopts the forecasting-based approach, while the LogBERT model uses the reconstruction-based approach. We focus on log data that consists of sequences of log sentences. A key factor affecting the effec- tiveness of …

Content truncated to 500 words