Digital Operatives at SemEval-2018 Task 8: Using dependency features for malware NLP

Proceedings of the 12th International Workshop on Semantic Evaluation (SemEval-2018), pages 894–897 New Orleans, Louisiana, June 5–6, 2018. ©2018 Association for Computational Linguistics Digital Operatives at SemEval-2018 Task 8: Using dependency features for malware NLP Chris Brew Digital Operatives chris.brew@digitaloperatives.com Abstract The four sub-tasks of SecureNLP build to- wards a capability for quickly highlighting critical information from malware reports, such as the specific actions taken by a mal- ware sample. Digital Operatives (DO) sub- mitted to sub-tasks 1 and 2, using standard text analysis technology (text classification for sub-task 1, and a CRF for sub-task 2). Perfor- mance is broadly competitive with other sub- mitted systems on sub-task 1 and weak on sub- task 2. The annotation guidelines for the in- termediate sub-tasks create a linkage to the fi- nal task, which is both an annotation challenge and a potentially useful feature of the task. The methods DO chose do not attempt to make use of this linkage, which may be a missed oppor- tunity. This motivates a post-hoc error analy- sis. It appears that the annotation task is very hard, and that in some cases both deep con- ceptual knowledge and substantial surround- ing context are needed in order to correctly classify sentences. 1 Introduction The SecureNLP challenge is motivated by (Lim et al., 2017) and further described in (Phandi et al., 2018), it aims to provide automation for malware analysts who might otherwise be overwhelmed by the task of finding key data in long threat reports. The annotation guidelines used to create the task ask analysts to include actions carried out by the malware, but exclude actions carried out by the hu- man designers of the malware. These actions are related to the MAEC cybersecurity ontology (Kir- illov et al., 2010). The guidelines include one sub- stantial caveat: As a general guide [for a positive an- notation], the sentence should imply a particular malware action or capability, with reference to the list of attribute la- bels. [i.e. the MAEC labels] Sub-task 1 calls for a determination of relevance or irrelevance to malware activity on a per-sentence basis. However a number of issues make this diffi- cult. See detail in (Lim et al., 2017). First, it is not obvious what to do when a sentence describes mal- ware activity but does not fit in with any MAEC category. Second, the distinction between things done by the malware and things done (or intended) by its designers is not easy to maintain. We describe the systems that we built for tasks 1 and 2 and use them to conduct ablation studies and error analyses. 2 Digital Operatives Systems The Digital Operatives submission used spaCy (Honnibal and Johnson, 2015). generate features for each token, then aggregatedto the features from the whole sentence. To estimate performance, we used 5-fold cross-validation on the combined training and development sets. As an example, consider the word ”ago” in the sentence: ”A few days ago we detected a watering hole campaign in a website owned by one …

Content truncated to 500 words