Villani at SemEval-2018 Task 8: Semantic Extraction from Cybersecurity Reports using Representation Learning

Proceedings of the 12th International Workshop on Semantic Evaluation (SemEval-2018), pages 885–889 New Orleans, Louisiana, June 5–6, 2018. ©2018 Association for Computational Linguistics Villani at SemEval-2018 Task 8: Semantic Extraction from Cybersecurity Reports using Representation Learning Pablo Loyola, Kugamoorthy Gajananan, Yuji Watanabe and Fumiko Satoh IBM Research Tokyo Tokyo, Japan {e57095,gajan,muew,sfumiko}@jp.ibm.com Abstract In this paper, we describe our proposal for the task of Semantic Extraction from Cybersecu- rity Reports. The goal is to explore if natu- ral language processing methods can provide relevant and actionable knowledge to con- tribute to better understand malicious behav- ior. Our method consists of an attention-based Bi-LSTM which achieved competitive perfor- mance of 0.57 for the Subtask 1. In the due process we also present ablation studies across multiple embeddings and their level of rep- resentation and also report the strategies we used to mitigate the extreme imbalance be- tween classes. 1 Introduction Cybersecurity represents one of the most com- prehensive and challenging tasks to tackle from a data-driven perspective. It is inherently technical, covering field such as networking and program- ming languages, but at the same time, it considers human aspects, such as intent, trust and strategy among benign and malicious agents (Buczak and Guven, 2016). This rich mixture makes it an ideal playground for machine learning, to extract patterns and char- acterize the interaction between the different set of actors involved. Moreover, as we can col- lect large amounts of data from security related sources, such as trace logs and reports, the level of generalization that machine learning methods could achieve could increase. Nevertheless, sev- eral challenges also emerge such as noise, lack structure, unavailability of annotated sources and a characteristic class imbalance when data is la- beled. Therefore, for machine learning to be con- sidered useful in a cybersecurity context, it must provide robust and reliable results, overcoming the aforementioned issues. In that sense, how we represent the data plays a key role, as it is known that in any machine learn- ing setting, different feature representations yield to different results, entangling different explana- tory factors of variation on the data (Bengio et al., 2013). We are interested in study the tradeoff be- tween the use of hand crafted features the process of automatically learning feature representations. In that sense, the present SemEval Task 8 (Phandi et al., 2018) represents a relevant sce- nario to test several hypotheses in the context of a controlled semantic extraction competition. The dataset, provided by Lim et al. (2017), contains over 6,800 labeled sentences from 39 malware re- ports. From the subtasks, we focused on the first one, as it provides compact goal to assess a proof of concept. Our approach consists on the use of an at- tention based LSTM-based recurrent architecture (Hochreiter and Schmidhuber, 1997; Luong et al., 2015) which is capable of learning sentence level feature representations at both character and to- ken level. Additionally, we prepend an embedding layer from which pretrained feature vectors can be associated to the tokens. Given the …

Content truncated to 500 words