Introducing a New Dataset for Event Detection in Cybersecurity Texts

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 5381–5390, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 5381 Introducing a New Dataset for Event Detection in Cybersecurity Texts Hieu Man Duc Trong1∗, Duc Trong Le1∗, Amir Pouran Ben Veyseh2∗, Thuat Nguyen1∗ and Thien Huu Nguyen2† 1 Hanoi University of Science and Technology, Hanoi, Vietnam 2 Department of Computer and Information Science, University of Oregon, Eugene, OR 97403, USA {hieu.mdt161530,duc.lt161082,thuat.nh163964}@sis.hust.edu.vn {apouranb,thien}@cs.uoregon.edu Abstract Detecting cybersecurity events is necessary to keep us informed about the fast growing num- ber of such events reported in text. In this work, we focus on the task of event detection (ED) to identify event trigger words for the cybersecurity domain. In particular, to facil- itate the future research, we introduce a new dataset for this problem, characterizing the manual annotation for 30 important cybersecu- rity event types and a large dataset size to de- velop deep learning models. Comparing to the prior datasets for this task, our dataset involves more event types and supports the modeling of document-level information to improve the performance. We perform extensive evalua- tion with the current state-of-the-art methods for ED on the proposed dataset. Our exper- iments reveal the challenges of cybersecurity ED and present many research opportunities in this area for the future work. 1 Introduction With the proliferation of cyber technologies (i.e., social networks, Internet of Things) in our daily life, the frequency of cyberattacks and cybercrimes is also rapidly increasing, potentially imposing se- rious threats to our cyber activities. Cybersecurity has thus become an important field for which a large amount of text data would be produced to report and discuss various aspects of cyber vulner- abilities. The expected sheer amount of this type of data calls for automatic techniques to analyze the cybersecurity text and extract useful knowledge. Among others, these techniques can help to detect the trends of the cyberattacks for better policy mak- ing or populate cybersecurity knowledge bases for automatic reasoning systems. In this work, we examine Information Extraction technologies (IE) in Natural Language Processing ∗The first four authors contribute equally to this paper. †Corresponding author. (NLP) as a promising candidate for the knowledge extraction task from cybersecurity text. In particu- lar, we focus on Event Detection (ED), an impor- tant task in IE that seeks to identify trigger words of specified types of events in text (Ahn, 2006; Ji and Grishman, 2008). ED is an actively studied task in IE where deep learning models have been the dominant approach to deliver the state-of-the-art performance (Nguyen and Grishman, 2015; Chen et al., 2015). For instance, consider the sentence: Remote attackers to completely takeover player accounts just by tricking users into clicking an un- suspectable link. An ED system for cybersecurity texts should be able to identify “takeover” as an event trigger word of the event type ATTACK.User Compromise in this case. In order to enable the application of the ED methods in the cybersecurity domain, a crucial re- quirement has …

Content truncated to 500 words