Flytxt_NTNU at SemEval-2018 Task 8: Identifying and Classifying Malware Text Using Conditional Random Fields and Naïve Bayes Classifiers

Proceedings of the 12th International Workshop on Semantic Evaluation (SemEval-2018), pages 890–893 New Orleans, Louisiana, June 5–6, 2018. ©2018 Association for Computational Linguistics Flytxt NTNU at SemEval-2018 Task 8: Identifying and Classifying Malware Text Using Conditional Random Fields and Na¨ıve Bayes Classifiers Utpal Kumar Sikdar1, Biswanath Barik2 and Bj¨orn Gamb¨ack2 1Flytxt, Thiruvananthapuram, India 2Department of Computer Science, NTNU, Norway utpal.sikdar@gmail.com {biswanath.barik,gamback}@ntnu.no Abstract Cybersecurity risks such as malware threaten the personal safety of users, but to identify malware text is a major challenge. The pa- per proposes a supervised learning approach to identifying malware sentences given a doc- ument (subTask1 of SemEval 2018, Task 8), as well as to classifying malware tokens in the sentences (subTask2). The approach achieved good results, ranking second of twelve partic- ipants for both subtasks, with F-scores of 57% for subTask1 and 28% for subTask2. 1 Introduction Malware is a major problem in the digital world. Recently, Lim et al. (2017) addressed the mal- ware threat by creating a new database of malware texts. In addition, they constructed different mod- els for identifying and classifying malware sen- tences, and discussed the outstanding challenges. Sutskever et al. (2016) also pointed to cybersecu- rity defense as a key area because of its long-term impact on society. Still, there have been very few efforts addressing the problem. Many cybersecu- rity agencies such as Cylance (Gross, 2016) and Symantec (DiMaggio, 2015) have collected large repositories of malware-related online texts. The diversity of those texts shows that identifying mal- ware is quite challenging. The organizers of SemEval 2018, Task 8 de- fined four subtasks for Semantic Extraction from CybersecUrity REports using NLP, SecureNLP (Phandi et al., 2018). The paper outlines a su- pervised approach to the first two subtasks, on malware sentence and token identification, respec- tively. In subTask1, given a sentence, the systems need to predict whether the sentence is relevant for inferring the malware’s actions and capabili- ties. For this subtask, two machine learning clas- sifiers were used, Na¨ıve Bayes (Rish, 2001) and Total Malware Data Sents Tokens Sents Tokens Train 9,435 231,180 2,204 12,165 Dev 1,213 32,029 79 459 Test 618 13,080 90 453 Table 1: Malware dataset statistics Conditional Random Fields (CRF, Lafferty et al., 2001), as well as a combination of the two models. In subTask2, the systems should predict and classify malware tokens in the sentences into three different categories, namely Action, Entity, and Modifier. A CRF-based classifier was used also for the second subtask. The paper is organized as follows: The datasets are presented in Section 2. The malware sentence identification is described in Section 3, while the token label malware identification is described in Section 4. Results are presented in Section 5, with system comparison and error analysis reported in Sections 6 and 7, respectively. Section 8 addresses future work and concludes. 2 Datasets The SecureNLP shared task organizers provided three different datasets: training, development and test sets. The statistics of the datasets are reported in Table 1, with the total number of …

Content truncated to 500 words