DarkBERT: A Language Model for the Dark Side of the Internet

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 7515–7533 July 9-14, 2023 ©2023 Association for Computational Linguistics DarkBERT: A Language Model for the Dark Side of the Internet Youngjin Jin1 Eugene Jang2 Jian Cui2 Jin-Woo Chung2 Yongjae Lee2 Seungwon Shin1 1KAIST, Daejeon, South Korea 2S2W Inc., Seongnam, South Korea 1{ijinjin,claude}@kaist.ac.kr 2{genesith,geeoon19,jwchung,lee}@s2w.inc Abstract Recent research has suggested that there are clear differences in the language used in the Dark Web compared to that of the Surface Web. As studies on the Dark Web commonly require textual analysis of the domain, language mod- els specific to the Dark Web may provide valu- able insights to researchers. In this work, we introduce DarkBERT, a language model pre- trained on Dark Web data. We describe the steps taken to filter and compile the text data used to train DarkBERT to combat the extreme lexical and structural diversity of the Dark Web that may be detrimental to building a proper rep- resentation of the domain. We evaluate Dark- BERT and its vanilla counterpart along with other widely used language models to validate the benefits that a Dark Web domain specific model offers in various use cases. Our evalua- tions show that DarkBERT outperforms current language models and may serve as a valuable resource for future research on the Dark Web. 1 Introduction The Dark Web is a subset of the Internet that is not indexed by web search engines such as Google and is inaccessible through a standard web browser. To access the Dark Web, specialized overlay network applications such as Tor (The Onion Router) (Din- gledine et al., 2004) are required. Tor also hosts hidden services (onion services) — web services in which the client and the server IP addresses are hidden from each other (Biryukov et al., 2013). This sense of identity obscurity provided to the Dark Web users comes with a catch; many of the underground activities prevalent in the Dark Web are immoral/illegal in nature, ranging from content hosting such as data leaks to drug sales (Al Nabki et al., 2017; Jin et al., 2022). As such, the pop- ularity of the Dark Web as a platform of choice for malicious activities has garnered interest from researchers and security experts alike. To handle the ever-changing landscape of mod- ern cyber threats, cybersecurity experts and re- searchers have started to employ natural language processing (NLP) methods. Gaining evidence- based knowledge such as indicators of compro- mise (IOC) to mitigate emerging threats is an inte- gral part of modern cybersecurity known as cyber threat intelligence (CTI) (Liao et al., 2016; Bromi- ley, 2016), and modern NLP tools have become an indispensable part of CTI research. As such, the use of NLP techniques has also been extended to the Dark Web (Jin et al., 2022; Yoon et al., 2019; Choshen et al., 2019; Al Nabki et al., 2017; Al- Nabki et al., 2019; Yuan et al., 2018). The con- tinued exploitation of the Dark Web …

Content truncated to 500 words