A Real-World Data Resource of Complex Sensitive Sentences Based on Documents from the Monsanto Trial

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 1258–1267 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 1258 A Real-World Data Resource of Complex Sensitive Sentences Based on Documents from the Monsanto Trial Jan Neerbek1,2, Morten Eskildsen1, Peter Dolog3, Ira Assent1 1Department of Computer Science, Aarhus University, Aarhus, Denmark 2Alexandra Institute, Aarhus, Denmark 3 Department of Computer Science, Aalborg University, Aalborg, Denmark jan.neerbek@cs.au.dk,morten@moddi.dk,dolog@cs.aau.dk,ira@cs.au.dk Abstract In this work we present a corpus for the evaluation of sensitive information detection approaches that addresses the need for real world sensitive information for empirical studies. Our sentence corpus contains different notions of complex sensitive information that corre- spond to different aspects of concern in a current trial of the Monsanto company. This paper describes the annotations process, where we both employ human annotators and furthermore create automatically inferred labels regarding technical, legal and informal communication within and with employees of Monsanto, drawing on a classification of documents by lawyers involved in the Monsanto court case. We release corpus of high quality sentences and parse trees with these two types of labels on sentence level. We characterize the sensitive information via several representative sensitive information detection models, in particular both keyword- based (n-gram) approaches and recent deep learning models, namely, recurrent neural networks (LSTM) and recursive neural networks (RecNN). Data and code are made publicly available. Keywords: Corpus (Creation, Annotation, etc.), Statistical and Machine Learning Methods, Document Classification, Text categorisation 1. Introduction Sensitive information detection addresses the problem of identifying (parts of) text documents that are considered sensitive in a particular application context. Sensitive infor- mation detection is of great importance in a number of ap- plications, where unintended leak of sensitive information may incur severe negative consequences for individuals, businesses or authorities. In a study from 2017 Poneman Institute and IBM find that the average cost of a breach of sensitive information is $3.6 million in total, for detection, escalation, notification, and after-the-fact response (Pone- man Institute, 2017; Poneman Institute, 2009). Sensitive information detection has been studied in the Natural Language Processing and Machine Learning re- search communities (Neerbek et al., 2018; S´anchez and Batet, 2017; S´anchez and Batet, 2016; Berardi et al., 2015; Gollins et al., 2014; Grechanik et al., 2014; Hart et al., 2011; Chow et al., 2008). In this work we focus on sensi- tive information detection considered as a form of text clas- sification, where the goal is to predict whether a sentence contains sensitive information. Here we consider sensitive information to be domain specific and the definition in 4 different datasets in this work is given by labels from do- main experts. A distinguishing feature of sensitive information detection with respect to traditional text classification is that we are interested not only in the content (topics and entities) but also in context (Gollins et al., 2014). In this work we fol- low (Gollins et al., 2014) and focus on context within a sentence. A limiting factor in research and …

Content truncated to 500 words