Identifying Products in Online Cybercrime Marketplaces: A Dataset for Fine-grained Domain Adaptation

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2598–2607 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Identifying Products in Online Cybercrime Marketplaces: A Dataset for Fine-grained Domain Adaptation Greg Durrett UT Austin gdurrett@cs.utexas.edu Jonathan K. Kummerfeld University of Michigan jkummerf@umich.edu Taylor Berg-Kirkpatrick Carnegie Mellon University tberg@cs.cmu.edu Rebecca S. Portnoff UC Berkeley rsportnoff@cs.berkeley.edu Sadia Afroz ICSI, UC Berkeley sadia@icsi.berkeley.edu Damon McCoy NYU mccoy@nyu.edu Kirill Levchenko UC San Diego klevchen@cs.ucsd.edu Vern Paxson ICSI, UC Berkeley vern@berkeley.edu Abstract One weakness of machine-learned NLP models is that they typically perform poorly on out-of-domain data. In this work, we study the task of identifying products being bought and sold in on- line cybercrime forums, which exhibits particularly challenging cross-domain ef- fects. We formulate a task that represents a hybrid of slot-filling information extrac- tion and named entity recognition and an- notate data from four different forums. Each of these forums constitutes its own “fine-grained domain” in that the forums cover different market sectors with differ- ent properties, even though all forums are in the broad domain of cybercrime. We characterize these domain differences in the context of a learning-based system: su- pervised models see decreased accuracy when applied to new forums, and stan- dard techniques for semi-supervised learn- ing and domain adaptation have limited ef- fectiveness on this data, which suggests the need to improve these techniques. We release a dataset of 1,938 annotated posts from across the four forums.1 1 Introduction NLP can be extremely useful for enabling scien- tific inquiry, helping us to quickly and efficiently understand large corpora, gather evidence, and test hypotheses (Bamman et al., 2013; O’Connor et al., 1Dataset and code to train models available at https://evidencebasedsecurity.org/forums/ TITLE: [ buy ] Backconnect bot BODY: Looking for a solid backconnect bot . If you know of anyone who codes them please let me know (a) File 0-initiator4856 TITLE: Exploit cleaning ? BODY: Have some Exploits i need fud . (b) File 0-initiator10815 Figure 1: Example posts and annotations from Darkode, with annotated product tokens under- lined. The second example exhibits jargon (fud means “fully undetectable”), nouns that could be a product in other contexts (Exploit), and multiple lexically-distinct descriptions of a single service. Note that these posts are much shorter than the av- erage Darkode post (61.5 words). 2013). One domain for which automated analy- sis is particularly useful is Internet security: re- searchers obtain large amounts of text data perti- nent to active threats or ongoing cybercriminal ac- tivity, for which the ability to rapidly characterize that text and draw conclusions can reap major ben- efits (Krebs, 2013a,b). However, conducting auto- matic analysis is difficult because this data is out- of-domain for conventional NLP models, which harms the performance of both discrete models (McClosky et al., 2010) and deep models (Zhang et al., 2017). Not only that, we show that data from one cybercrime forum is even out of domain with respect to another cybercrime forum, making this data especially challenging. In …

Content truncated to 500 words