The Creation and Analysis of a Website Privacy Policy Corpus

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1330–1340, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics The Creation and Analysis of a Website Privacy Policy Corpus Shomir Wilson1, Florian Schaub1, Aswarth Abhilash Dara1, Frederick Liu1, Sushain Cherivirala1, Pedro Giovanni Leon2, Mads Schaarup Andersen1, Sebastian Zimmeck3, Kanthashree Mysore Sathyendra1, N. Cameron Russell4, Thomas B. Norton4, Eduard Hovy1, Joel Reidenberg4 and Norman Sadeh1 1Carnegie Mellon University, School of Computer Science, Pittsburgh, PA, USA 2Stanford University, Center for Internet and Society, Stanford, CA, USA 3Columbia University, Department of Computer Science, New York, NY, USA 4Fordham University, Law School, New York, NY, USA shomir@cs.cmu.edu, sadeh@cs.cmu.edu Abstract Website privacy policies are often ignored by Internet users, because these docu- ments tend to be long and difficult to un- derstand. However, the significance of pri- vacy policies greatly exceeds the attention paid to them: these documents are binding legal agreements between website opera- tors and their users, and their opaqueness is a challenge not only to Internet users but also to policy regulators. One proposed al- ternative to the status quo is to automate or semi-automate the extraction of salient de- tails from privacy policy text, using a com- bination of crowdsourcing, natural lan- guage processing, and machine learning. However, there has been a relative dearth of datasets appropriate for identifying data practices in privacy policies. To remedy this problem, we introduce a corpus of 115 privacy policies (267K words) with man- ual annotations for 23K fine-grained data practices. We describe the process of us- ing skilled annotators and a purpose-built annotation tool to produce the data. We provide findings based on a census of the annotations and show results toward au- tomating the annotation procedure. Fi- nally, we describe challenges and oppor- tunities for the research community to use this corpus to advance research in both pri- vacy and language technologies. 1 Introduction Privacy policies written in natural language are a nearly pervasive feature of websites and mo- bile applications. The “notice and choice” le- gal regimes of many countries require that web- site operators post a notice of how they gather and process users’ information. In theory, users then choose whether to accept those practices or to abstain from using the website or service. In practice, however, the average Internet user strug- gles to understand the contents of privacy poli- cies (McDonald and Cranor, 2008) and generally does not read them (Federal Trade Commission, 2012; President’s Concil of Advisors on Science and Technology, 2014). This disconnect between Internet users and the data practices that affect them has led to the assessment that the notice and choice model is ineffective in the status quo (Rei- denberg et al., 2015b; Cate, 2010). Thus, an opening exists for language technolo- gies to help “bridge the gap” between privacy poli- cies in their current form and representations that serve the needs of Internet users. Such a bridge would also serve unmet needs of policy regula- tors, who do not have …

Content truncated to 500 words