Automatic Identification of Indicators of Compromise using Neural-Based Sequence Labelling

Automatic Identification of Indicators of Compromise using Neural-Based Sequence Labelling Shengping Zhou Zi Long Lianzhi Tan Hao Guo Mobile Internet Group, Tencent Hans Laser Technology Centre , Shennan Ave No.9988, Nanshan District, Shenzhen City, Guangdong Province, 518057, China Abstract Indicators of Compromise (IOCs) are arti- facts observed on a network or in an operat- ing system that can be utilized to indicate a computer intrusion and detect cyber-attacks in an early stage. Thus, they exert an impor- tant role in the field of cybersecurity. How- ever, state-of-the-art IOCs detection systems rely heavily on hand-crafted features with ex- pert knowledge of cybersecurity, and require a large amount of supervised training corpora to train an IOC classifier. In this paper, we pro- pose using a neural-based sequence labelling model to identify IOCs automatically from re- ports on cybersecurity without expert knowl- edge of cybersecurity. Our work is the first to apply an end-to-end sequence labelling to the task in IOCs identification. By using an at- tention mechanism and several token spelling features, we find that the proposed model is capable of identifying the low frequency IOCs from long sentences contained in cybersecu- rity reports. Experiments show that the pro- posed model outperforms other sequence la- belling models, achieving over 88% average F1-score. 1 Introduction Indicators of Compromise (IOCs) are forensic arti- facts that are used as signs when a system has been compromised by an attacker or infected with a par- ticular piece of malware. To be specific, IOCs are composed of some combinations of virus signatures, IPs, URLs or domain names of botnets, MD5 hashes of attack files, etc. They are frequently described in cybersecurity reports, much of which are written in unstructured text, describing attack tactics, tech- nique and procedures, and can be utilized for early detection of future attack attempts by using intru- sion detection systems and antivirus software. With the rapid evolvement of cyber threats, the IOC data are produced at a high volume and velocity every day, which makes it increasingly hard for human to gather and manage them. A number of systems are proposed to help discover and gather malicious information and IOCs from various types of data sources (Zhu and Dumitras, 2016; Liao et al., 2016; Husari et al., 2017; Huang et al., 2017; Kwon et al., 2017; Zhu and Dumitras, 2018). However, most of them identify IOCs by using human-crafted features that heavily rely on specific language knowledge such as dependency structure, and they often have to be pre-defined by experts in the field of the cyberse- curity. Furthermore, they need a large amount of an- notated data used as the training data to train an IOC classifier. Those training data are frequently diffi- cult to be crowed-sourced, because non-experts can hardly distinguish IOCs from those non-malicious IPs or URLs. Thus, it is a time-consuming and labo- rious task to construct such a system. In this paper, we propose using an end-to-end neural sequence labelling model to fully automate the process of IOCs identification. …

Content truncated to 500 words